Jul 16 17:34:38.516609 ip-10-0-132-31 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.863756    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868420    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868432    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868437    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868440    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868443    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868446    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868450    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:38.876849 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868453    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868456    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868459    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868462    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868465    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868468    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868470    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868473    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868476    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868479    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868484    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868488    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868491    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868494    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868497    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868501    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868504    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868507    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868509    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868512    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:38.877656 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868515    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868518    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868520    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868523    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868525    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868528    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868531    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868533    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868536    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868538    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868541    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868545    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868549    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868552    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868555    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868558    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868561    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868564    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868566    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:38.878275 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868571    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868573    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868576    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868579    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868581    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868584    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868586    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868589    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868592    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868594    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868597    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868600    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868602    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868605    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868607    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868610    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868612    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868617    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868621    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868624    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:38.878883 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868627    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868629    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868632    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868634    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868637    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868640    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868643    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868646    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868649    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868651    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868654    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868657    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868659    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868662    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868665    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868667    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868670    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868673    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868675    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:38.879469 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868678    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868680    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868683    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868685    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868688    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868691    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868693    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868696    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868698    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868701    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868703    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868705    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868708    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.868711    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869415    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869422    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869426    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869430    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869433    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869436    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:38.880021 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869439    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869442    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869444    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869447    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869450    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869452    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869455    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869458    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869461    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869463    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869466    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869468    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869471    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869473    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869476    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869479    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869482    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869484    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869487    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:38.880588 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869489    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869492    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869495    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869498    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869500    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869503    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869505    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869508    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869513    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869517    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869520    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869524    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869527    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869530    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869533    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869536    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869539    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869542    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869544    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:38.881099 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869547    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869550    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869554    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869560    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869563    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869566    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869568    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869571    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869573    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869576    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869579    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869581    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869584    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869586    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869588    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869592    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869596    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869599    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869602    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:38.881651 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869605    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869608    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869610    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869613    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869615    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869618    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869620    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869623    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869625    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869627    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869630    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869632    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869635    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869637    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869639    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869642    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869647    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869650    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869653    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869655    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:38.882416 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869658    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869661    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869664    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869666    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869669    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869671    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869673    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869676    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869679    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869681    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869683    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869686    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869689    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869691    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869695    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.869698    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870066    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870071    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870073    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870076    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:38.882993 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870079    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870081    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870084    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870087    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870089    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870091    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870094    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870096    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870101    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870105    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870109    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870112    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870116    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870118    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870121    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870123    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870126    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870128    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870131    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870134    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:38.883627 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870136    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870139    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870141    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870144    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870147    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870149    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870153    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870156    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870158    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870161    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870164    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870166    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870169    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870171    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870174    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870176    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870179    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870181    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870184    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870186    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:38.884142 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870190    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870193    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870196    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870198    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870201    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870203    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870206    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870208    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870211    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870213    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870215    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870218    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870221    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870223    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870226    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870228    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870230    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870235    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870240    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:38.884751 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870243    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870246    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870248    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870251    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870253    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870256    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870258    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870261    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870279    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870282    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870284    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870287    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870290    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870294    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870296    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870299    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870302    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870305    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870308    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:38.885363 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870310    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870313    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870315    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870318    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870320    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870323    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870325    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870329    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870332    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870335    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870338    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870341    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870346    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870349    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870351    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870354    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.870357    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870424    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870431    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870438    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870442    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 17:34:38.970075 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870446    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 17:34:38.933150 ip-10-0-132-31 systemd[1]: Started Kubernetes Kubelet.
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870449    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870453    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870456    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870460    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870464    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870468    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870471    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870476    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870479    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870481    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870484    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870487    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870490    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870494    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870497    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870500    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870514    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870517    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870521    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870524    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870527    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870531    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870536    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 17:34:38.971035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870539    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870541    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870545    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870548    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870551    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870554    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870558    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870560    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870563    2579 flags.go:64] FLAG: --enable-server="true"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870566    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870571    2579 flags.go:64] FLAG: --event-burst="100"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870574    2579 flags.go:64] FLAG: --event-qps="50"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870577    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870581    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870584    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870588    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870591    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870594    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870597    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870600    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870603    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870606    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870609    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870611    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870614    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 17:34:38.972159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870617    2579 flags.go:64] FLAG: --feature-gates=""
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870620    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870623    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870626    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870629    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870632    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870636    2579 flags.go:64] FLAG: --help="false"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870639    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-31.ec2.internal"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870642    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870645    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870648    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870651    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870655    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870658    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870661    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870663    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870666    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870669    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870672    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870675    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870679    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870681    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870685    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870687    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 17:34:38.973441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870690    2579 flags.go:64] FLAG: --lock-file=""
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870693    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870696    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870699    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870704    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870707    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870710    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870713    2579 flags.go:64] FLAG: --logging-format="text"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870715    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870719    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870721    2579 flags.go:64] FLAG: --manifest-url=""
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870724    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870728    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870732    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870737    2579 flags.go:64] FLAG: --max-pods="110"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870740    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870743    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870746    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870748    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870751    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870754    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870757    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870764    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870767    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870770    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 17:34:38.974490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870773    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870776    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870782    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870785    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870789    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870791    2579 flags.go:64] FLAG: --port="10250"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870795    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870797    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bb5c7ff4f2fcc572"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870801    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870804    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870807    2579 flags.go:64] FLAG: --register-node="true"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870810    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870814    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870816    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870819    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870822    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870825    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870828    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870831    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870834    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870837    2579 flags.go:64] FLAG: --runonce="false"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870839    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870843    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870846    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870849    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 17:34:38.975705 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870852    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870855    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870857    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870861    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870863    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870866    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870869    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870871    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870875    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870878    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870881    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870887    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870890    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870893    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870896    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870899    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870902    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870905    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870908    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870911    2579 flags.go:64] FLAG: --v="2"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870915    2579 flags.go:64] FLAG: --version="false"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870919    2579 flags.go:64] FLAG: --vmodule=""
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870923    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.870926    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871034    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:38.976521 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871037    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871040    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871043    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871046    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871049    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871052    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871055    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871057    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871060    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871062    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871065    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871068    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871071    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871073    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871075    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871078    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871080    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871085    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871087    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:38.977240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871090    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871092    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871095    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871097    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871100    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871102    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871105    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871108    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871111    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871113    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871116    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871118    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871121    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871123    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871126    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871129    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871131    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871135    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871137    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871140    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:38.977954 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871142    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871145    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871147    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871150    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871154    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871157    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871160    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871164    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871168    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871171    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871175    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871177    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871180    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871182    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871185    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871188    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871190    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871193    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871196    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:38.978673 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871199    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871202    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871205    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871207    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871210    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871213    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871215    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871218    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871220    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871222    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871226    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871229    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871231    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871233    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871236    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871238    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871242    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871246    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871249    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871252    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:38.979202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871255    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871257    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871260    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871279    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871282    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871286    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871290    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871293    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871296    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871299    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871302    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871305    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871307    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871310    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871313    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871315    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871318    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871321    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871323    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:38.979966 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.871326    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.871334    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.881337    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.881355    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.881948    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.881958    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882017    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882023    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882026    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882029    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882032    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882035    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882038    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882041    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882044    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:39.119932 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882046    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882049    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882051    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882054    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882057    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882060    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882063    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882065    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882068    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882071    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882073    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882076    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882079    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882082    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882085    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882088    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882090    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882094    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882099    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:39.144773 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882102    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882111    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882114    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882117    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882120    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882123    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882126    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882129    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882132    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882136    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882139    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882142    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882144    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882147    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882150    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882153    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882156    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882159    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882161    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882164    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:39.145429 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882167    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882169    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882172    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882174    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882177    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882179    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882183    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882185    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882188    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882191    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882193    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882196    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882198    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882201    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882209    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882212    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882215    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882217    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882220    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882222    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:39.146150 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882225    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882227    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882230    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882232    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882234    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882237    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882240    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882242    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882245    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882248    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882251    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882253    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882256    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882259    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882261    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882277    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882281    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882284    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882287    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:39.146787 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882290    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882293    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882295    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882298    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882301    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882304    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882307    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882309    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882321    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882324    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882326    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882329    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.882335    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882477    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882482    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:39.147422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882485    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882488    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882491    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882494    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882497    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882499    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882511    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882514    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882517    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882519    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882522    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882524    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882527    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882530    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882532    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882535    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882538    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882540    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882543    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882545    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:39.148035 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882549    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882551    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882554    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882557    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882559    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882562    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882570    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882574    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882577    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882580    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882582    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882585    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882587    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882590    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882593    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882595    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882598    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882601    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882604    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:39.148666 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882606    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882609    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882611    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882614    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882616    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882619    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882621    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882624    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882627    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882629    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882632    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882634    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882637    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882640    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882643    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882645    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882648    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882650    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882653    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882656    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:39.149474 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882666    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882669    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882672    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882675    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882679    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882681    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882684    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882687    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882689    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882691    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882694    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882697    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882699    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882702    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882705    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882707    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882710    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882713    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882715    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882718    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:39.150077 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882720    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882723    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882725    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882728    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882731    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882733    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882736    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882738    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882741    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882743    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882746    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882749    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882751    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882761    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882764    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882767    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882771    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:39.150875 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:38.882776    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.882781    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.882909    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.884762    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.885829    2579 server.go:1019] "Starting client certificate rotation"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.885965    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.886591    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.888116    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.889885    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.903364    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.905243    2579 log.go:25] "Validated CRI v1 image API"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.905855    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 17:34:39.151487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.908348    2579 fs.go:135] Filesystem UUIDs: map[0e5c690e-f8f3-4905-aa67-64b8d6a3aa35:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 ce8e8992-d0bf-4d74-b898-384d5e69e26d:/dev/nvme0n1p4]
Jul 16 17:34:39.151960 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.908366    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 17:34:39.151960 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.913676    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.913891    2579 manager.go:216] Machine: {Timestamp:2026-07-16 17:34:38.912061051 +0000 UTC m=+0.295530976 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097355 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec295a04281c2ea8c42d95ec08e866fd SystemUUID:ec295a04-281c-2ea8-c42d-95ec08e866fd BootID:8ba4d7a9-2616-4c37-aceb-e4f6185f9be1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:f0:58:68:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:f0:58:68:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:f7:26:cd:dc:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.913993    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.914068    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.916670    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.916691    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-31.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.917238    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.917247    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.917255    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.917859    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.918781    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922368    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922509    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922544    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922556    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922574    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922585    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922597    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 17:34:39.152137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.922984    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.923004    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.924997    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927034    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927067    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927296    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927313    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927319    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927326    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927334    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927339    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927708    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927716    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927722    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927730    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927740    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.927750    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.928927    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.928936    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.931924    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-31.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.932170    2579 server.go:1268] "Started kubelet"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.932240    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.932391    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.932391    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.932442    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.932512    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.934075    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.936455    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.937148    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.940540    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.940896    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.941710    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943300    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943317    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943456    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943465    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943542    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943664    2579 factory.go:55] Registering systemd factory
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.943683    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.944824    2579 factory.go:153] Registering CRI-O factory
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.944843    2579 factory.go:223] Registration of the crio container factory successfully
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.944926    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.944951    2579 factory.go:103] Registering Raw factory
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.944966    2579 manager.go:1196] Started watching for new ooms in manager
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.945610    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c266s"
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.945775    2579 manager.go:349] Starting recovery of all containers
Jul 16 17:34:39.152798 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.946362    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-31.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.948574    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.946241    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-31.ec2.internal.18c2d59e74bb749a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-31.ec2.internal,UID:ip-10-0-132-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-31.ec2.internal,},FirstTimestamp:2026-07-16 17:34:38.932128922 +0000 UTC m=+0.315598848,LastTimestamp:2026-07-16 17:34:38.932128922 +0000 UTC m=+0.315598848,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-31.ec2.internal,}"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.955198    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-c266s"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.957539    2579 manager.go:354] Recovery completed
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.962664    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.964451    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.964574    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.964586    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.965163    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.965172    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.965186    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.965451    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.966057    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.966074    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.966094    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:38.967029    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-31.ec2.internal.18c2d59e76aa571a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-31.ec2.internal,UID:ip-10-0-132-31.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-31.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-31.ec2.internal,},FirstTimestamp:2026-07-16 17:34:38.96456169 +0000 UTC m=+0.348031615,LastTimestamp:2026-07-16 17:34:38.96456169 +0000 UTC m=+0.348031615,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-31.ec2.internal,}"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.967340    2579 policy_none.go:49] "None policy: Start"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.967363    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.967373    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:38.969406    2579 policy_none.go:47] "Start"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013009    2579 manager.go:337] "Starting Device Plugin manager"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.013036    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013045    2579 server.go:87] "Starting device plugin registration server"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013297    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013310    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013408    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013545    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.013553    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.013919    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.013966    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.095511    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.096685    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.096719    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.096739    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 17:34:39.154109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.096770    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.099316    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.114475    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.115358    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.115379    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.115388    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.115407    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.125476    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.125494    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-31.ec2.internal\": node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.155210 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.142528    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.203396    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"]
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.203486    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.204418    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.204445    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.204455    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.206805    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.206938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.206969    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.207542    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.207559    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.207584    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.207594    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.207568    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.207653    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.209928    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.209955    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.210701    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.210729    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.210741    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.242961    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.245341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba822384b1d7a38b02d50006563516f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal\" (UID: \"ba822384b1d7a38b02d50006563516f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.245364    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba822384b1d7a38b02d50006563516f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal\" (UID: \"ba822384b1d7a38b02d50006563516f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.245388    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56b83bb518a676ee4805f90256f69199-config\") pod \"kube-apiserver-proxy-ip-10-0-132-31.ec2.internal\" (UID: \"56b83bb518a676ee4805f90256f69199\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.255014    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-31.ec2.internal\" not found" node="ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.284306 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.259524    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-31.ec2.internal\" not found" node="ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.343361    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.345648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba822384b1d7a38b02d50006563516f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal\" (UID: \"ba822384b1d7a38b02d50006563516f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.345669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56b83bb518a676ee4805f90256f69199-config\") pod \"kube-apiserver-proxy-ip-10-0-132-31.ec2.internal\" (UID: \"56b83bb518a676ee4805f90256f69199\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.345686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba822384b1d7a38b02d50006563516f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal\" (UID: \"ba822384b1d7a38b02d50006563516f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.345728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ba822384b1d7a38b02d50006563516f7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal\" (UID: \"ba822384b1d7a38b02d50006563516f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.345754    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba822384b1d7a38b02d50006563516f7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal\" (UID: \"ba822384b1d7a38b02d50006563516f7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.345774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/56b83bb518a676ee4805f90256f69199-config\") pod \"kube-apiserver-proxy-ip-10-0-132-31.ec2.internal\" (UID: \"56b83bb518a676ee4805f90256f69199\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.456730 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.443623    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.612679 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.548962    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.612679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.557827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.612679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.561946    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:39.695102 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.650032    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.758999 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.750474    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.911745 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.851390    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:39.911745 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.887242    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 17:34:39.911745 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.887421    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 17:34:40.036563 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:39.952163    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-31.ec2.internal\" not found"
Jul 16 17:34:40.036563 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.959622    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:29:39 +0000 UTC" deadline="2028-03-30 16:51:15.008948062 +0000 UTC"
Jul 16 17:34:40.036563 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.959646    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14951h16m35.049304827s"
Jul 16 17:34:40.036563 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.974105    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:40.036563 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:39.986263    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:40.150984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.042386    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:40.150984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.058863    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:34:40.150984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.060431    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal"
Jul 16 17:34:40.150984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.072361    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:34:40.235435 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.235336    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:40.276995 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:40.276963    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba822384b1d7a38b02d50006563516f7.slice/crio-f5966b95c25a08b18f63675afee3dac2be7848264e2a494898a927f8f8e859bb WatchSource:0}: Error finding container f5966b95c25a08b18f63675afee3dac2be7848264e2a494898a927f8f8e859bb: Status 404 returned error can't find the container with id f5966b95c25a08b18f63675afee3dac2be7848264e2a494898a927f8f8e859bb
Jul 16 17:34:40.277786 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:40.277768    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56b83bb518a676ee4805f90256f69199.slice/crio-67bee1d0554221e083ed1724ef467e29a720076735fbd5b35b1175caa6b9a0bc WatchSource:0}: Error finding container 67bee1d0554221e083ed1724ef467e29a720076735fbd5b35b1175caa6b9a0bc: Status 404 returned error can't find the container with id 67bee1d0554221e083ed1724ef467e29a720076735fbd5b35b1175caa6b9a0bc
Jul 16 17:34:40.282605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.282586    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:34:40.924625 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.924592    2579 apiserver.go:52] "Watching apiserver"
Jul 16 17:34:40.936317 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.936287    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 17:34:40.936848 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.936807    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-rngnz","kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal","openshift-dns/node-resolver-254d8","openshift-image-registry/node-ca-wxrdp","openshift-multus/multus-l8rvw","openshift-multus/network-metrics-daemon-d5x6f","openshift-network-diagnostics/network-check-target-jxtmc","openshift-ovn-kubernetes/ovnkube-node-vm2hw","kube-system/konnectivity-agent-4rtn2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf","openshift-cluster-node-tuning-operator/tuned-ggsdl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal","openshift-multus/multus-additional-cni-plugins-pgzsj","openshift-network-operator/iptables-alerter-df5s8"]
Jul 16 17:34:40.939512 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.939484    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:40.939635 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:40.939570    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:40.941921 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.941892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.942220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.942196    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 17:34:40.944539 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.944509    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:40.948737 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.948099    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:40.949808 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.949783    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950523    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950548    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950625    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950698    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qjg4m\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950849    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 17:34:40.951287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.950904    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.951779 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.951520    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 17:34:40.951874 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.951834    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xm2dz\""
Jul 16 17:34:40.952588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.952565    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.952884 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.952866    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rzn8\""
Jul 16 17:34:40.953086 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.953070    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:40.953168 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:40.953140    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:40.955394 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.955372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:40.956306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.956266    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jks2b\""
Jul 16 17:34:40.956618 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.956597    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 17:34:40.958116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958089    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-os-release\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958214 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:40.958214 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958214 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958193    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45ktr\" (UniqueName: \"kubernetes.io/projected/813304a6-08e0-48ab-a660-f8acfe893c99-kube-api-access-45ktr\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958411 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:40.958210    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:40.958411 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958222    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-system-cni-dir\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958411 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-cni-binary-copy\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958411 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958316    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-hosts-file\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:40.958411 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzxm\" (UniqueName: \"kubernetes.io/projected/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-kube-api-access-qqzxm\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-tmp-dir\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxhsh\" (UniqueName: \"kubernetes.io/projected/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-kube-api-access-vxhsh\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-host\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-serviceca\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:40.958663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958639    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:40.959014 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958673    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-cnibin\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:40.959014 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.958862    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 17:34:40.959539 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.959477    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l55hw\""
Jul 16 17:34:40.959632 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.959560    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.959780 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.959762    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 17:34:40.960557 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.960536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:40.963617 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.963596    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.963790 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.963771    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:40.965660 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.965639    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.965935 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.965914    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jqjqk\""
Jul 16 17:34:40.966032 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.965987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 17:34:40.966277 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.966256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:40.968931 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.968907    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.969456 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.969432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:40.972131 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.972076    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.973246 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.972384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 17:34:40.973246 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.972551    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.973246 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.972718    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.973246 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.973051    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:34:40.975373 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.975300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-98kn6\""
Jul 16 17:34:40.975458 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.975400    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nczq4\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.975625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kv2zl\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.975866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.975919    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.976051    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.976069    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.976212    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 17:34:40.976420 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.976273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 17:34:40.999563 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:40.999540    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vg5bs"
Jul 16 17:34:41.007473 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.007452    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vg5bs"
Jul 16 17:34:41.045238 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.045214    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 17:34:41.059003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.058978    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-cni-bin\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.059123 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.059123 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-socket-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.059123 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-registration-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.059123 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059119    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-kubernetes\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-cni-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxhsh\" (UniqueName: \"kubernetes.io/projected/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-kube-api-access-vxhsh\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/441e47ae-e5d0-4faf-ae89-d8f798600e46-cni-binary-copy\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059292    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-slash\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-node-log\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-system-cni-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059461    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-cni-bin\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059723    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-kubelet\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-etc-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059779    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.059871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovnkube-config\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059854    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-tmp\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-k8s-cni-cncf-io\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-daemon-config\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-cni-netd\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.059976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-env-overrides\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-tmp-dir\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-host\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060082    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/641ec5e3-fe02-4a0c-ae35-04125798cee6-iptables-alerter-script\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060105    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-systemd\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060140    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-cni-multus\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d05dd297-4cd0-4616-8cc1-53d85de33013-agent-certs\") pod \"konnectivity-agent-4rtn2\" (UID: \"d05dd297-4cd0-4616-8cc1-53d85de33013\") " pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-kubelet-config\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-cnibin\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060468    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-tmp-dir\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060517    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45ktr\" (UniqueName: \"kubernetes.io/projected/813304a6-08e0-48ab-a660-f8acfe893c99-kube-api-access-45ktr\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.060723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060559    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-cnibin\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwxb\" (UniqueName: \"kubernetes.io/projected/641ec5e3-fe02-4a0c-ae35-04125798cee6-kube-api-access-pmwxb\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060617    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-netns\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060648    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-etc-kubernetes\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbf2f\" (UniqueName: \"kubernetes.io/projected/441e47ae-e5d0-4faf-ae89-d8f798600e46-kube-api-access-rbf2f\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060891    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-ovn\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-system-cni-dir\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.060982    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-sys\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr966\" (UniqueName: \"kubernetes.io/projected/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-kube-api-access-pr966\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061194    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-system-cni-dir\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysctl-conf\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061241    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-run\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-host\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061285    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-lib-modules\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-socket-dir-parent\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.061620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-dbus\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-run-netns\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-systemd\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovn-node-metrics-cert\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061600    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-sys-fs\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061638    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d05dd297-4cd0-4616-8cc1-53d85de33013-konnectivity-ca\") pod \"konnectivity-agent-4rtn2\" (UID: \"d05dd297-4cd0-4616-8cc1-53d85de33013\") " pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-var-lib-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-device-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-host\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061817    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-conf-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-systemd-units\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-hosts-file\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061910    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzxm\" (UniqueName: \"kubernetes.io/projected/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-kube-api-access-qqzxm\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2jfm\" (UniqueName: \"kubernetes.io/projected/096c8735-9c1a-477e-ac22-6730160e99ee-kube-api-access-v2jfm\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.061981    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-tuned\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062033    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-hosts-file\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.062571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062144    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-cnibin\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-os-release\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-log-socket\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062232    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovnkube-script-lib\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysconfig\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-kubelet\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062332    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-serviceca\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062484    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-os-release\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6ljh\" (UniqueName: \"kubernetes.io/projected/b72558f3-3a3d-480a-9540-c427f35bfc10-kube-api-access-t6ljh\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/641ec5e3-fe02-4a0c-ae35-04125798cee6-host-slash\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062583    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-modprobe-d\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-hostroot\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062640    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-multus-certs\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.063492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-cni-binary-copy\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.064269 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062754    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysctl-d\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.064269 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-var-lib-kubelet\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.064269 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062794    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-serviceca\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.064269 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.062830    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-776mh\" (UniqueName: \"kubernetes.io/projected/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-kube-api-access-776mh\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.075226 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.074166    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 17:34:41.075450 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.075217    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/813304a6-08e0-48ab-a660-f8acfe893c99-os-release\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.076049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.075943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/813304a6-08e0-48ab-a660-f8acfe893c99-cni-binary-copy\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.078901 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.078730    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzxm\" (UniqueName: \"kubernetes.io/projected/71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1-kube-api-access-qqzxm\") pod \"node-ca-wxrdp\" (UID: \"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1\") " pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.079912 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.079888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxhsh\" (UniqueName: \"kubernetes.io/projected/1bcb2707-bb60-4d1d-b477-bbe7cc04c710-kube-api-access-vxhsh\") pod \"node-resolver-254d8\" (UID: \"1bcb2707-bb60-4d1d-b477-bbe7cc04c710\") " pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.081171 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.081112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45ktr\" (UniqueName: \"kubernetes.io/projected/813304a6-08e0-48ab-a660-f8acfe893c99-kube-api-access-45ktr\") pod \"multus-additional-cni-plugins-pgzsj\" (UID: \"813304a6-08e0-48ab-a660-f8acfe893c99\") " pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.086765 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.086745    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:41.086853 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.086771    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:41.086853 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.086785    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:41.086976 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.086856    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:41.586824807 +0000 UTC m=+2.970294740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:41.104221 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.104176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal" event={"ID":"ba822384b1d7a38b02d50006563516f7","Type":"ContainerStarted","Data":"f5966b95c25a08b18f63675afee3dac2be7848264e2a494898a927f8f8e859bb"}
Jul 16 17:34:41.105943 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.105917    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal" event={"ID":"56b83bb518a676ee4805f90256f69199","Type":"ContainerStarted","Data":"67bee1d0554221e083ed1724ef467e29a720076735fbd5b35b1175caa6b9a0bc"}
Jul 16 17:34:41.165976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.165945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6ljh\" (UniqueName: \"kubernetes.io/projected/b72558f3-3a3d-480a-9540-c427f35bfc10-kube-api-access-t6ljh\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:41.166100 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.165993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/641ec5e3-fe02-4a0c-ae35-04125798cee6-host-slash\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.166100 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-modprobe-d\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166100 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166051    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-hostroot\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166100 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-multus-certs\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysctl-d\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/641ec5e3-fe02-4a0c-ae35-04125798cee6-host-slash\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-var-lib-kubelet\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166176    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-hostroot\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-776mh\" (UniqueName: \"kubernetes.io/projected/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-kube-api-access-776mh\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-cni-bin\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-socket-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-registration-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-kubernetes\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-cni-bin\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.166353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-cni-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166447    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-modprobe-d\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-multus-certs\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-cni-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-var-lib-kubelet\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166577    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-registration-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166477    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/441e47ae-e5d0-4faf-ae89-d8f798600e46-cni-binary-copy\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166636    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-kubernetes\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166643    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysctl-d\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166679    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-slash\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166682    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-socket-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-node-log\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.166743    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-slash\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166758    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-system-cni-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.166908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-node-log\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166804    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.166821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:41.666794478 +0000 UTC m=+3.050264392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-system-cni-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-cni-bin\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-cni-bin\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166896    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-kubelet\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166922    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-etc-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166960    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovnkube-config\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166992    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-tmp\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-kubelet\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167018    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-k8s-cni-cncf-io\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.166998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/441e47ae-e5d0-4faf-ae89-d8f798600e46-cni-binary-copy\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167044    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-daemon-config\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167070    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-cni-netd\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-etc-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.167817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-env-overrides\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-k8s-cni-cncf-io\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-cni-netd\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167172    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/641ec5e3-fe02-4a0c-ae35-04125798cee6-iptables-alerter-script\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-systemd\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-cni-multus\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167249    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d05dd297-4cd0-4616-8cc1-53d85de33013-agent-certs\") pod \"konnectivity-agent-4rtn2\" (UID: \"d05dd297-4cd0-4616-8cc1-53d85de33013\") " pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-kubelet-config\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwxb\" (UniqueName: \"kubernetes.io/projected/641ec5e3-fe02-4a0c-ae35-04125798cee6-kube-api-access-pmwxb\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-netns\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-etc-kubernetes\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167417    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbf2f\" (UniqueName: \"kubernetes.io/projected/441e47ae-e5d0-4faf-ae89-d8f798600e46-kube-api-access-rbf2f\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-ovn\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167488    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-sys\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167524    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-env-overrides\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167560    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr966\" (UniqueName: \"kubernetes.io/projected/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-kube-api-access-pr966\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysctl-conf\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.168710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167640    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-run\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-lib-modules\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167695    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-ovn\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-socket-dir-parent\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167746    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-dbus\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-sys\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-run-netns\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167793    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/641ec5e3-fe02-4a0c-ae35-04125798cee6-iptables-alerter-script\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167800    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-systemd\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-kubelet-config\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167828    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovn-node-metrics-cert\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-sys-fs\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167865    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-run\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d05dd297-4cd0-4616-8cc1-53d85de33013-konnectivity-ca\") pod \"konnectivity-agent-4rtn2\" (UID: \"d05dd297-4cd0-4616-8cc1-53d85de33013\") " pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-var-lib-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.167944    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-socket-dir-parent\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.169701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-etc-kubernetes\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167979    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-run-netns\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-device-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.168004    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:41.667986856 +0000 UTC m=+3.051456784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysctl-conf\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-cni-multus\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168046    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-device-dir\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168057    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-host\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-daemon-config\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-host\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-systemd\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168128    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-sys-fs\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovnkube-config\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168153    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-run-netns\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.167913    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-systemd\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-dbus\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-run-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.170639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-conf-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168267    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-multus-conf-dir\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-systemd-units\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168276    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-lib-modules\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168310    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-var-lib-openvswitch\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-systemd-units\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2jfm\" (UniqueName: \"kubernetes.io/projected/096c8735-9c1a-477e-ac22-6730160e99ee-kube-api-access-v2jfm\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-tuned\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168408    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-cnibin\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-os-release\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-log-socket\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168490    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovnkube-script-lib\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysconfig\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-kubelet\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168607    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-cnibin\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d05dd297-4cd0-4616-8cc1-53d85de33013-konnectivity-ca\") pod \"konnectivity-agent-4rtn2\" (UID: \"d05dd297-4cd0-4616-8cc1-53d85de33013\") " pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.171493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-sysconfig\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-host-var-lib-kubelet\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/441e47ae-e5d0-4faf-ae89-d8f798600e46-os-release\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168849    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-host-run-ovn-kubernetes\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-log-socket\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.168882    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/096c8735-9c1a-477e-ac22-6730160e99ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.169252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovnkube-script-lib\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.169894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-tmp\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.170813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d05dd297-4cd0-4616-8cc1-53d85de33013-agent-certs\") pod \"konnectivity-agent-4rtn2\" (UID: \"d05dd297-4cd0-4616-8cc1-53d85de33013\") " pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.170980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-ovn-node-metrics-cert\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.172432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.171139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-etc-tuned\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.182885 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.182644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2jfm\" (UniqueName: \"kubernetes.io/projected/096c8735-9c1a-477e-ac22-6730160e99ee-kube-api-access-v2jfm\") pod \"aws-ebs-csi-driver-node-lnkhf\" (UID: \"096c8735-9c1a-477e-ac22-6730160e99ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.183322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.183056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6ljh\" (UniqueName: \"kubernetes.io/projected/b72558f3-3a3d-480a-9540-c427f35bfc10-kube-api-access-t6ljh\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:41.183322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.183183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwxb\" (UniqueName: \"kubernetes.io/projected/641ec5e3-fe02-4a0c-ae35-04125798cee6-kube-api-access-pmwxb\") pod \"iptables-alerter-df5s8\" (UID: \"641ec5e3-fe02-4a0c-ae35-04125798cee6\") " pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.185975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.185928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-776mh\" (UniqueName: \"kubernetes.io/projected/e13a3966-19c8-4b8d-a1f2-404dbbc0baca-kube-api-access-776mh\") pod \"tuned-ggsdl\" (UID: \"e13a3966-19c8-4b8d-a1f2-404dbbc0baca\") " pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.186362 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.186331    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr966\" (UniqueName: \"kubernetes.io/projected/7c1955ab-f9aa-431c-bbcf-f234b5f25a0f-kube-api-access-pr966\") pod \"ovnkube-node-vm2hw\" (UID: \"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f\") " pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.186470 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.186448    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbf2f\" (UniqueName: \"kubernetes.io/projected/441e47ae-e5d0-4faf-ae89-d8f798600e46-kube-api-access-rbf2f\") pod \"multus-l8rvw\" (UID: \"441e47ae-e5d0-4faf-ae89-d8f798600e46\") " pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.258747 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.258715    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pgzsj"
Jul 16 17:34:41.270689 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:41.270660    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod813304a6_08e0_48ab_a660_f8acfe893c99.slice/crio-c371408b43a6440d9911d5f76d4b6b48bc581f77e5e7c28c8525bb2b49bb9ee6 WatchSource:0}: Error finding container c371408b43a6440d9911d5f76d4b6b48bc581f77e5e7c28c8525bb2b49bb9ee6: Status 404 returned error can't find the container with id c371408b43a6440d9911d5f76d4b6b48bc581f77e5e7c28c8525bb2b49bb9ee6
Jul 16 17:34:41.271397 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.271377    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-254d8"
Jul 16 17:34:41.282480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.282430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wxrdp"
Jul 16 17:34:41.291429 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.291404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l8rvw"
Jul 16 17:34:41.299261 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.299233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:34:41.310071 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.310048    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-df5s8"
Jul 16 17:34:41.320091 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.319933    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf"
Jul 16 17:34:41.328862 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.328841    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggsdl"
Jul 16 17:34:41.335769 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.335681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:34:41.407424 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.407395    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:41.673614 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.673527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:41.673614 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.673578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:41.673843 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673714    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:41.673843 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:41.673748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:41.673843 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:42.673782681 +0000 UTC m=+4.057252614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:41.673843 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673725    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:41.673843 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673836    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:41.674139 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673852    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:41.674139 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673870    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:41.674139 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673887    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:42.673875319 +0000 UTC m=+4.057345231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:41.674139 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:41.673920    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:42.673903175 +0000 UTC m=+4.057373109 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:42.011143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.011024    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:41 +0000 UTC" deadline="2028-01-29 07:13:59.50214179 +0000 UTC"
Jul 16 17:34:42.011143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.011092    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13477h39m17.491054571s"
Jul 16 17:34:42.097994 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.097961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:42.098162 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.097976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:42.098162 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.098107    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:42.098300 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.098164    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:42.108244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.108217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerStarted","Data":"c371408b43a6440d9911d5f76d4b6b48bc581f77e5e7c28c8525bb2b49bb9ee6"}
Jul 16 17:34:42.406913 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.406881    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05dd297_4cd0_4616_8cc1_53d85de33013.slice/crio-78f4b8b300c9b0f784fe635809b7341caa0a0486c8bee7f5fd1a61d0d80570a3 WatchSource:0}: Error finding container 78f4b8b300c9b0f784fe635809b7341caa0a0486c8bee7f5fd1a61d0d80570a3: Status 404 returned error can't find the container with id 78f4b8b300c9b0f784fe635809b7341caa0a0486c8bee7f5fd1a61d0d80570a3
Jul 16 17:34:42.408325 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.408280    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1955ab_f9aa_431c_bbcf_f234b5f25a0f.slice/crio-1d963a297f59885a60b1a6783e7cb418b3726f91119c9aa9e77ed50223f8aa3a WatchSource:0}: Error finding container 1d963a297f59885a60b1a6783e7cb418b3726f91119c9aa9e77ed50223f8aa3a: Status 404 returned error can't find the container with id 1d963a297f59885a60b1a6783e7cb418b3726f91119c9aa9e77ed50223f8aa3a
Jul 16 17:34:42.411722 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.411681    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13a3966_19c8_4b8d_a1f2_404dbbc0baca.slice/crio-7496fb98e5b533ffa6c888f8a8d953a587918d49d137a911ca2c275722a6cf96 WatchSource:0}: Error finding container 7496fb98e5b533ffa6c888f8a8d953a587918d49d137a911ca2c275722a6cf96: Status 404 returned error can't find the container with id 7496fb98e5b533ffa6c888f8a8d953a587918d49d137a911ca2c275722a6cf96
Jul 16 17:34:42.412889 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.412861    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641ec5e3_fe02_4a0c_ae35_04125798cee6.slice/crio-6c483c49616c86abb14162b32c285be443c10238368193fb8c19af0355edb32c WatchSource:0}: Error finding container 6c483c49616c86abb14162b32c285be443c10238368193fb8c19af0355edb32c: Status 404 returned error can't find the container with id 6c483c49616c86abb14162b32c285be443c10238368193fb8c19af0355edb32c
Jul 16 17:34:42.414878 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.414445    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod096c8735_9c1a_477e_ac22_6730160e99ee.slice/crio-055a635ab56b483884e58ede627193e14c1fbb736104e243710ec691ed59f41b WatchSource:0}: Error finding container 055a635ab56b483884e58ede627193e14c1fbb736104e243710ec691ed59f41b: Status 404 returned error can't find the container with id 055a635ab56b483884e58ede627193e14c1fbb736104e243710ec691ed59f41b
Jul 16 17:34:42.415505 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.415482    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod441e47ae_e5d0_4faf_ae89_d8f798600e46.slice/crio-f6bb29af1e4fa1665593a3b53531181f504a1cae2ba1e0931367b6d3bada45f3 WatchSource:0}: Error finding container f6bb29af1e4fa1665593a3b53531181f504a1cae2ba1e0931367b6d3bada45f3: Status 404 returned error can't find the container with id f6bb29af1e4fa1665593a3b53531181f504a1cae2ba1e0931367b6d3bada45f3
Jul 16 17:34:42.418050 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.417398    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f18ce5_721a_48b2_a9d9_3aa00b1ff7a1.slice/crio-99233267a9b970d71da17ccb8b816885a0b7a72370763a7eefee8a6fd249eb0c WatchSource:0}: Error finding container 99233267a9b970d71da17ccb8b816885a0b7a72370763a7eefee8a6fd249eb0c: Status 404 returned error can't find the container with id 99233267a9b970d71da17ccb8b816885a0b7a72370763a7eefee8a6fd249eb0c
Jul 16 17:34:42.418380 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:34:42.418355    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcb2707_bb60_4d1d_b477_bbe7cc04c710.slice/crio-d00734d74defbdda47cc456c1d53e41a5bb4612b4fb16599671387b538095a79 WatchSource:0}: Error finding container d00734d74defbdda47cc456c1d53e41a5bb4612b4fb16599671387b538095a79: Status 404 returned error can't find the container with id d00734d74defbdda47cc456c1d53e41a5bb4612b4fb16599671387b538095a79
Jul 16 17:34:42.681933 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.681717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:42.682039 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.681884    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:42.682099 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682055    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:44.682037067 +0000 UTC m=+6.065506981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:42.682099 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682071    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:42.682200 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682123    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:44.682107403 +0000 UTC m=+6.065577339 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:42.682200 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.681989    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:42.682200 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:42.682179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:42.682462 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682263    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:42.682462 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682276    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:42.682462 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682285    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:42.682462 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:42.682340    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:44.682315216 +0000 UTC m=+6.065785131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:43.011603 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.011532    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:41 +0000 UTC" deadline="2028-04-21 06:18:09.783316906 +0000 UTC"
Jul 16 17:34:43.011603 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.011577    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15468h43m26.771745439s"
Jul 16 17:34:43.100069 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.099860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:43.100069 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:43.100024    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:43.118333 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.118239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8rvw" event={"ID":"441e47ae-e5d0-4faf-ae89-d8f798600e46","Type":"ContainerStarted","Data":"f6bb29af1e4fa1665593a3b53531181f504a1cae2ba1e0931367b6d3bada45f3"}
Jul 16 17:34:43.124561 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.124525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-df5s8" event={"ID":"641ec5e3-fe02-4a0c-ae35-04125798cee6","Type":"ContainerStarted","Data":"6c483c49616c86abb14162b32c285be443c10238368193fb8c19af0355edb32c"}
Jul 16 17:34:43.127615 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.127586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"1d963a297f59885a60b1a6783e7cb418b3726f91119c9aa9e77ed50223f8aa3a"}
Jul 16 17:34:43.136418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.136388    2579 generic.go:356] "Generic (PLEG): container finished" podID="ba822384b1d7a38b02d50006563516f7" containerID="284fc3eff57c564cec20f569bfaa084a3e8cfad1da2c79d06ff7ba068e099688" exitCode=0
Jul 16 17:34:43.136550 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.136503    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal" event={"ID":"ba822384b1d7a38b02d50006563516f7","Type":"ContainerDied","Data":"284fc3eff57c564cec20f569bfaa084a3e8cfad1da2c79d06ff7ba068e099688"}
Jul 16 17:34:43.142404 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.142331    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf" event={"ID":"096c8735-9c1a-477e-ac22-6730160e99ee","Type":"ContainerStarted","Data":"055a635ab56b483884e58ede627193e14c1fbb736104e243710ec691ed59f41b"}
Jul 16 17:34:43.146411 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.146358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggsdl" event={"ID":"e13a3966-19c8-4b8d-a1f2-404dbbc0baca","Type":"ContainerStarted","Data":"7496fb98e5b533ffa6c888f8a8d953a587918d49d137a911ca2c275722a6cf96"}
Jul 16 17:34:43.155661 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.155635    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rtn2" event={"ID":"d05dd297-4cd0-4616-8cc1-53d85de33013","Type":"ContainerStarted","Data":"78f4b8b300c9b0f784fe635809b7341caa0a0486c8bee7f5fd1a61d0d80570a3"}
Jul 16 17:34:43.165602 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.165565    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal" event={"ID":"56b83bb518a676ee4805f90256f69199","Type":"ContainerStarted","Data":"320a12a1bf95682d8ffdffffe91202d469adc9ca888419d6ec4483673b971675"}
Jul 16 17:34:43.167977 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.167940    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-254d8" event={"ID":"1bcb2707-bb60-4d1d-b477-bbe7cc04c710","Type":"ContainerStarted","Data":"d00734d74defbdda47cc456c1d53e41a5bb4612b4fb16599671387b538095a79"}
Jul 16 17:34:43.169816 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:43.169791    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wxrdp" event={"ID":"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1","Type":"ContainerStarted","Data":"99233267a9b970d71da17ccb8b816885a0b7a72370763a7eefee8a6fd249eb0c"}
Jul 16 17:34:44.097029 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.097000    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:44.097506 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.097141    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:44.097506 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.097363    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:44.097506 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.097459    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:44.181092 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.180838    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal" event={"ID":"ba822384b1d7a38b02d50006563516f7","Type":"ContainerStarted","Data":"67f384e6df2233841ce22a84cdf2f5906b131ba0b0077dbae686f7e9e9724006"}
Jul 16 17:34:44.202627 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.202536    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-31.ec2.internal" podStartSLOduration=4.202514469 podStartE2EDuration="4.202514469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:43.187359919 +0000 UTC m=+4.570829858" watchObservedRunningTime="2026-07-16 17:34:44.202514469 +0000 UTC m=+5.585984403"
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.700317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.700378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:44.700424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.700549    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.700611    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.700591994 +0000 UTC m=+10.084061909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.701014    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.701032    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.701045    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.701090    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.701075223 +0000 UTC m=+10.084545138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.701149    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:44.701257 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:44.701181    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:48.701170896 +0000 UTC m=+10.084640816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:45.097632 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:45.097548    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:45.098072 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:45.097698    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:46.098637 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:46.097936    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:46.098637 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:46.098072    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:46.098637 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:46.098447    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:46.098637 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:46.098536    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:47.097956 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:47.097923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:47.098133 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:47.098085    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:48.097671 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:48.097638    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:48.098126 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.097766    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:48.098126 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:48.098114    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:48.098233 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.098203    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:48.731925 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:48.731885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:48.732096 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:48.731962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:48.732096 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:48.732012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:48.732203 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732159    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:48.732203 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732182    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:48.732203 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732194    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:48.732375 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732250    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:34:56.732232707 +0000 UTC m=+18.115702636 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:48.732690 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732670    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:48.732765 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732730    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:34:56.732715649 +0000 UTC m=+18.116185577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:48.732825 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732789    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:48.732825 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:48.732823    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:56.732810471 +0000 UTC m=+18.116280398 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:49.097973 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:49.097886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:49.098489 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:49.098001    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:50.097145 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:50.097098    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:50.097367 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:50.097154    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:50.097367 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:50.097241    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:50.097476 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:50.097366    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:51.097499 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:51.097461    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:51.097946 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:51.097612    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:52.097668 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:52.097636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:52.098095 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:52.097636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:52.098095 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:52.097755    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:52.098095 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:52.097851    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:53.097693 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:53.097662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:53.098146 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:53.097796    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:54.097738 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:54.097706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:54.098160 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:54.097707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:54.098160 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:54.097821    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:54.098160 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:54.097885    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:55.097309 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:55.097258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:55.097489 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:55.097424    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:56.097498 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:56.097454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:56.097498 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:56.097500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:56.098013 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.097586    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:56.098013 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.097745    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:56.795423 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:56.795327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:56.795423 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:56.795398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:56.795437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795529    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795539    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795560    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795575    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795539    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795610    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:12.795590857 +0000 UTC m=+34.179060786 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795630    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:35:12.795619651 +0000 UTC m=+34.179089567 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:56.795659 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:56.795646    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:12.795637201 +0000 UTC m=+34.179107113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:57.097677 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:57.097595    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:57.098086 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:57.097745    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:34:58.097153 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:58.097116    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:34:58.097329 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:58.097246    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:34:58.097329 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:58.097292    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:34:58.097420 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:58.097400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:34:59.100766 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:34:59.100513    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:34:59.100766 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:34:59.100686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:00.096987 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:00.096948    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:00.097140 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:00.096971    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:00.097140 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:00.097085    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:00.097140 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:00.097130    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:01.097887 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.097837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:01.098224 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:01.097964    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:01.217150 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.216874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerStarted","Data":"9bf8d68a4b3bc476c611d31a55d834fe13c3bc184e5fa1b16884b7d882d1b0d2"}
Jul 16 17:35:01.222544 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.222393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-4rtn2" event={"ID":"d05dd297-4cd0-4616-8cc1-53d85de33013","Type":"ContainerStarted","Data":"711d84d8e5c0c2556db466c1d1a8c3f7a58d711c4a4e47d06688de7d73b826c6"}
Jul 16 17:35:01.225292 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.224140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wxrdp" event={"ID":"71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1","Type":"ContainerStarted","Data":"f2588f68bb140ab4d8a28c60f7cd0fdd466996a7c3fcdfe57ffb9e89941ed438"}
Jul 16 17:35:01.282809 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.282769    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-31.ec2.internal" podStartSLOduration=21.282753374 podStartE2EDuration="21.282753374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:44.203585726 +0000 UTC m=+5.587055663" watchObservedRunningTime="2026-07-16 17:35:01.282753374 +0000 UTC m=+22.666223307"
Jul 16 17:35:01.374046 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.373940    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-4rtn2" podStartSLOduration=4.028891794 podStartE2EDuration="22.373916429s" totalImagesPullingTime="18.345024635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:01.341846709 +0000 UTC m=+22.725316645" watchObservedRunningTime="2026-07-16 17:35:01.373916429 +0000 UTC m=+22.757386383"
Jul 16 17:35:01.374197 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:01.374163    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ggsdl" podStartSLOduration=3.749151934 podStartE2EDuration="22.374153059s" totalImagesPullingTime="18.625001125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:01.373728147 +0000 UTC m=+22.757198094" watchObservedRunningTime="2026-07-16 17:35:01.374153059 +0000 UTC m=+22.757622993"
Jul 16 17:35:02.097356 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.097170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:02.097489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.097230    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:02.097489 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:02.097428    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:02.097566 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:02.097534    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:02.228354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.228322    2579 generic.go:356] "Generic (PLEG): container finished" podID="813304a6-08e0-48ab-a660-f8acfe893c99" containerID="9bf8d68a4b3bc476c611d31a55d834fe13c3bc184e5fa1b16884b7d882d1b0d2" exitCode=0
Jul 16 17:35:02.229201 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.228418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerDied","Data":"9bf8d68a4b3bc476c611d31a55d834fe13c3bc184e5fa1b16884b7d882d1b0d2"}
Jul 16 17:35:02.230397 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.230374    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf" event={"ID":"096c8735-9c1a-477e-ac22-6730160e99ee","Type":"ContainerStarted","Data":"222cfae79fc8b61ecb3e3fc4660a41b8725dee55cb1481f50ede3aa063076142"}
Jul 16 17:35:02.232035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.232009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggsdl" event={"ID":"e13a3966-19c8-4b8d-a1f2-404dbbc0baca","Type":"ContainerStarted","Data":"e2ea4fe82bdcf7e50e3193ea737fe8f3484a731a66232a6c4f997ad5a50c29cc"}
Jul 16 17:35:02.233611 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.233582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-254d8" event={"ID":"1bcb2707-bb60-4d1d-b477-bbe7cc04c710","Type":"ContainerStarted","Data":"1678c843995b8da098043533eb435a0c9cd1e56e8c2d7d320b7c24c26151f315"}
Jul 16 17:35:02.235155 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.235131    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l8rvw" event={"ID":"441e47ae-e5d0-4faf-ae89-d8f798600e46","Type":"ContainerStarted","Data":"488b81303feb20752e0e4fcacae4d2ecf9a48e8f2dd191f1f3001fdaa473b847"}
Jul 16 17:35:02.238257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238239    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:35:02.238730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238704    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c1955ab-f9aa-431c-bbcf-f234b5f25a0f" containerID="febdc026f0031c27ea20fa49513557e846cc877c6304b4fd72ca66df13a8777c" exitCode=1
Jul 16 17:35:02.238810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"3696f434ee23b6681a0cd52b030445d896c729ae02e02a9389347dd7c21d637a"}
Jul 16 17:35:02.238874 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"be108540cdf3473549338404da9a75010b306133c2661dd8e161fa9f4f971597"}
Jul 16 17:35:02.238874 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"1fcb50206f37a0eac57fc7fe997485839a70f342243e61a7b62d310cc9283876"}
Jul 16 17:35:02.238977 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238892    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"4d6cd4dd2198ccf93073d5e39b24ef8595f61696b493ba53dbc067039df05ba7"}
Jul 16 17:35:02.238977 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238958    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerDied","Data":"febdc026f0031c27ea20fa49513557e846cc877c6304b4fd72ca66df13a8777c"}
Jul 16 17:35:02.239053 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.238992    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"7be661b540c5d18fe2d24ff4cb51203a78a61fb65c5eda5c19be727a12960f09"}
Jul 16 17:35:02.275657 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.275614    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wxrdp" podStartSLOduration=9.278646517 podStartE2EDuration="23.275600904s" totalImagesPullingTime="13.996954387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:01.40170481 +0000 UTC m=+22.785174743" watchObservedRunningTime="2026-07-16 17:35:02.275600904 +0000 UTC m=+23.659070841"
Jul 16 17:35:02.344762 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.344697    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-254d8" podStartSLOduration=4.744230327 podStartE2EDuration="23.34467812s" totalImagesPullingTime="18.600447793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:02.308788046 +0000 UTC m=+23.692257979" watchObservedRunningTime="2026-07-16 17:35:02.34467812 +0000 UTC m=+23.728148055"
Jul 16 17:35:02.344948 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.344890    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l8rvw" podStartSLOduration=4.548014544 podStartE2EDuration="23.344882772s" totalImagesPullingTime="18.796868228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:02.344454784 +0000 UTC m=+23.727924720" watchObservedRunningTime="2026-07-16 17:35:02.344882772 +0000 UTC m=+23.728352711"
Jul 16 17:35:02.547180 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.547142    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:35:02.547696 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.547679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:35:02.942485 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:02.942460    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 17:35:03.026396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.026312    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:35:02.94247796Z","UUID":"75dcc3cf-c6d6-4967-8bba-447392f9a738","Handler":null,"Name":"","Endpoint":""}
Jul 16 17:35:03.027779 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.027764    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 17:35:03.027842 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.027795    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 17:35:03.097571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.097514    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:03.097679 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:03.097615    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:03.241636 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.241604    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-df5s8" event={"ID":"641ec5e3-fe02-4a0c-ae35-04125798cee6","Type":"ContainerStarted","Data":"0748f6274f827cd8504652b6a0d5550e48a070aa92dd9a524c821ae077e7b517"}
Jul 16 17:35:03.243529 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.243502    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf" event={"ID":"096c8735-9c1a-477e-ac22-6730160e99ee","Type":"ContainerStarted","Data":"e491b2e46978ba6e95cf81e20c5ad7a8094be3cb60e5767b5e44dcaee16f075a"}
Jul 16 17:35:03.243917 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.243893    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:35:03.244306 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.244291    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-4rtn2"
Jul 16 17:35:03.260784 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:03.260741    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-df5s8" podStartSLOduration=5.920924455 podStartE2EDuration="24.260728549s" totalImagesPullingTime="18.339804094s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:03.260209612 +0000 UTC m=+24.643679546" watchObservedRunningTime="2026-07-16 17:35:03.260728549 +0000 UTC m=+24.644198483"
Jul 16 17:35:04.097628 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:04.097367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:04.097782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:04.097416    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:04.097782 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:04.097668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:04.097782 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:04.097760    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:04.249313 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:04.249283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:35:04.249735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:04.249710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"e5efc5ac70def5653109789d25223f00b05635a3d93d01da6b522552550fe935"}
Jul 16 17:35:05.097293 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:05.097245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:05.097474 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:05.097426    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:05.254840 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:05.254800    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf" event={"ID":"096c8735-9c1a-477e-ac22-6730160e99ee","Type":"ContainerStarted","Data":"5a3e5af5931a6268fa0b047f642bc347d3dd8ddf3f2a6167763c741a958c7407"}
Jul 16 17:35:05.298950 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:05.298895    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lnkhf" podStartSLOduration=4.740139556 podStartE2EDuration="26.298877511s" totalImagesPullingTime="21.558737955s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:05.298204552 +0000 UTC m=+26.681674486" watchObservedRunningTime="2026-07-16 17:35:05.298877511 +0000 UTC m=+26.682347445"
Jul 16 17:35:06.097885 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:06.097856    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:06.098053 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:06.097881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:06.098053 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:06.097961    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:06.098146 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:06.098086    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:07.097448 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.097104    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:07.097448 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:07.097427    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:07.262787 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.262759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:35:07.263142 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.263121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"3e53c9759ffc147c7931232ba5acb9b567255695b0d7440388b2bf0d37dbd632"}
Jul 16 17:35:07.263395 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.263373    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:35:07.263521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.263436    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:35:07.263690 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.263657    2579 scope.go:117] "RemoveContainer" containerID="febdc026f0031c27ea20fa49513557e846cc877c6304b4fd72ca66df13a8777c"
Jul 16 17:35:07.281103 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:07.281086    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:35:08.097452 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.097425    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:08.097452 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.097455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:08.097853 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:08.097516    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:08.097853 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:08.097658    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:08.268950 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.268889    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:35:08.269354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.269321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" event={"ID":"7c1955ab-f9aa-431c-bbcf-f234b5f25a0f","Type":"ContainerStarted","Data":"a1560e37be1c721c102109374e4e86c0c3fcfe291ee24219cfec2c0471b5acfe"}
Jul 16 17:35:08.269559 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.269544    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:35:08.271446 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.271421    2579 generic.go:356] "Generic (PLEG): container finished" podID="813304a6-08e0-48ab-a660-f8acfe893c99" containerID="ea60d9f59eb21c4d751274b31fd47ae23fc6d4e883071d5ce70a5791caa21b2e" exitCode=0
Jul 16 17:35:08.271569 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.271466    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerDied","Data":"ea60d9f59eb21c4d751274b31fd47ae23fc6d4e883071d5ce70a5791caa21b2e"}
Jul 16 17:35:08.287010 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.286986    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:35:08.354561 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:08.354490    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw" podStartSLOduration=10.675359937 podStartE2EDuration="29.354466785s" totalImagesPullingTime="18.679106848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:08.31071052 +0000 UTC m=+29.694180455" watchObservedRunningTime="2026-07-16 17:35:08.354466785 +0000 UTC m=+29.737936721"
Jul 16 17:35:09.098214 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.098186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:09.098550 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:09.098289    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:09.519337 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.519089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxtmc"]
Jul 16 17:35:09.519452 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.519375    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:09.519495 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:09.519477    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:09.522579 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.522547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rngnz"]
Jul 16 17:35:09.522734 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.522635    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:09.522790 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:09.522726    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:09.523159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.523139    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d5x6f"]
Jul 16 17:35:09.523259 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:09.523226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:09.523383 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:09.523363    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:11.097244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:11.097212    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:11.097905 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:11.097213    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:11.097905 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:11.097222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:11.097905 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:11.097337    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:11.097905 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:11.097421    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:11.097905 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:11.097489    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:11.279736 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:11.279703    2579 generic.go:356] "Generic (PLEG): container finished" podID="813304a6-08e0-48ab-a660-f8acfe893c99" containerID="047b9daa962a4131454c63258a8ce3b500aa70dcf475197b996477ce2383cf35" exitCode=0
Jul 16 17:35:11.279862 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:11.279742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerDied","Data":"047b9daa962a4131454c63258a8ce3b500aa70dcf475197b996477ce2383cf35"}
Jul 16 17:35:12.284697 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:12.284667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerStarted","Data":"5bff5e4d40f36a676a7298150f3542da897a72769988d2b8e008c3d0e9d95628"}
Jul 16 17:35:12.817438 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:12.817407    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:12.817438 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:12.817446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:12.817472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817543    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817556    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817569    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817577    2579 projected.go:196] Error preparing data for projected volume kube-api-access-kbbx4 for pod openshift-network-diagnostics/network-check-target-jxtmc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817598    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret podName:ec9208e8-03c0-4bf2-85f6-7e1470d0da5f nodeName:}" failed. No retries permitted until 2026-07-16 17:35:44.817584648 +0000 UTC m=+66.201054560 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret") pod "global-pull-secret-syncer-rngnz" (UID: "ec9208e8-03c0-4bf2-85f6-7e1470d0da5f") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817546    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4 podName:e544922c-45ee-4046-9bac-4c20b78fb57d nodeName:}" failed. No retries permitted until 2026-07-16 17:35:44.817606164 +0000 UTC m=+66.201076076 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kbbx4" (UniqueName: "kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4") pod "network-check-target-jxtmc" (UID: "e544922c-45ee-4046-9bac-4c20b78fb57d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:35:12.817639 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:12.817624    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:44.817618643 +0000 UTC m=+66.201088555 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:35:13.097290 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.097194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:13.097290 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.097226    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:13.097290 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.097194    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:13.097655 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:13.097318    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxtmc" podUID="e544922c-45ee-4046-9bac-4c20b78fb57d"
Jul 16 17:35:13.097655 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:13.097450    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5x6f" podUID="b72558f3-3a3d-480a-9540-c427f35bfc10"
Jul 16 17:35:13.097655 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:13.097524    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rngnz" podUID="ec9208e8-03c0-4bf2-85f6-7e1470d0da5f"
Jul 16 17:35:13.288857 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.288822    2579 generic.go:356] "Generic (PLEG): container finished" podID="813304a6-08e0-48ab-a660-f8acfe893c99" containerID="5bff5e4d40f36a676a7298150f3542da897a72769988d2b8e008c3d0e9d95628" exitCode=0
Jul 16 17:35:13.289194 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.288870    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerDied","Data":"5bff5e4d40f36a676a7298150f3542da897a72769988d2b8e008c3d0e9d95628"}
Jul 16 17:35:13.974325 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.974091    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-31.ec2.internal" event="NodeReady"
Jul 16 17:35:13.974476 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:13.974418    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 17:35:14.113704 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.113670    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qtjkf"]
Jul 16 17:35:14.117339 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.117318    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5blf5"]
Jul 16 17:35:14.117461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.117430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.120078 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.120049    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.126975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.126817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 17:35:14.126975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.126827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 17:35:14.126975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.126817    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 17:35:14.126975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.126887    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-27svs\""
Jul 16 17:35:14.126975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.126903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 17:35:14.126975 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.126935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2d8t\""
Jul 16 17:35:14.147033 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.147013    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 17:35:14.157738 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.157711    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5blf5"]
Jul 16 17:35:14.177561 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.177537    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qtjkf"]
Jul 16 17:35:14.227394 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.227294    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec720fcf-f94b-433d-907b-a361b69fa425-config-volume\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.227394 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.227322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skq7b\" (UniqueName: \"kubernetes.io/projected/ec720fcf-f94b-433d-907b-a361b69fa425-kube-api-access-skq7b\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.227394 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.227358    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec720fcf-f94b-433d-907b-a361b69fa425-tmp-dir\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.227609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.227464    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.227609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.227544    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5b3a7e-65e1-4ef1-ab25-9944805aac4f-cert\") pod \"ingress-canary-5blf5\" (UID: \"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f\") " pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.227609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.227585    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8skrx\" (UniqueName: \"kubernetes.io/projected/0a5b3a7e-65e1-4ef1-ab25-9944805aac4f-kube-api-access-8skrx\") pod \"ingress-canary-5blf5\" (UID: \"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f\") " pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.328019 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.327986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.328449 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.328048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5b3a7e-65e1-4ef1-ab25-9944805aac4f-cert\") pod \"ingress-canary-5blf5\" (UID: \"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f\") " pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.328449 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.328113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8skrx\" (UniqueName: \"kubernetes.io/projected/0a5b3a7e-65e1-4ef1-ab25-9944805aac4f-kube-api-access-8skrx\") pod \"ingress-canary-5blf5\" (UID: \"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f\") " pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.328449 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.328135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec720fcf-f94b-433d-907b-a361b69fa425-config-volume\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.328449 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.328153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skq7b\" (UniqueName: \"kubernetes.io/projected/ec720fcf-f94b-433d-907b-a361b69fa425-kube-api-access-skq7b\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.328449 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:14.328171    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:14.328449 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:14.328259    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:14.828234034 +0000 UTC m=+36.211703958 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:14.328713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.328546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec720fcf-f94b-433d-907b-a361b69fa425-tmp-dir\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.329009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.328982    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec720fcf-f94b-433d-907b-a361b69fa425-tmp-dir\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.329125 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.329084    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec720fcf-f94b-433d-907b-a361b69fa425-config-volume\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.332935 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.332908    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0a5b3a7e-65e1-4ef1-ab25-9944805aac4f-cert\") pod \"ingress-canary-5blf5\" (UID: \"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f\") " pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.342251 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.342225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8skrx\" (UniqueName: \"kubernetes.io/projected/0a5b3a7e-65e1-4ef1-ab25-9944805aac4f-kube-api-access-8skrx\") pod \"ingress-canary-5blf5\" (UID: \"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f\") " pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.343652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.343631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skq7b\" (UniqueName: \"kubernetes.io/projected/ec720fcf-f94b-433d-907b-a361b69fa425-kube-api-access-skq7b\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.435280 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.435225    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5blf5"
Jul 16 17:35:14.607159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.607120    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5blf5"]
Jul 16 17:35:14.611346 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:35:14.611320    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a5b3a7e_65e1_4ef1_ab25_9944805aac4f.slice/crio-304ad3c37a976a977c294edca2f9a6351c192e3f23229da3827ff5833e452899 WatchSource:0}: Error finding container 304ad3c37a976a977c294edca2f9a6351c192e3f23229da3827ff5833e452899: Status 404 returned error can't find the container with id 304ad3c37a976a977c294edca2f9a6351c192e3f23229da3827ff5833e452899
Jul 16 17:35:14.832617 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:14.832585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:14.832788 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:14.832743    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:14.832850 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:14.832815    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:15.832795798 +0000 UTC m=+37.216265722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:15.097942 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.097805    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:15.097942 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.097817    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:15.098154 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.098074    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:15.105765 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.105741    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28hgh\""
Jul 16 17:35:15.109490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.106911    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:35:15.109490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.107316    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:35:15.109490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.107531    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:35:15.109490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.107744    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4hnr\""
Jul 16 17:35:15.109490 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.107998    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:35:15.294957 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.294925    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5blf5" event={"ID":"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f","Type":"ContainerStarted","Data":"304ad3c37a976a977c294edca2f9a6351c192e3f23229da3827ff5833e452899"}
Jul 16 17:35:15.841209 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:15.841078    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:15.841689 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:15.841248    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:15.841689 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:15.841338    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:17.841316332 +0000 UTC m=+39.224786251 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:17.855185 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:17.854986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:17.855618 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:17.855137    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:17.855618 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:17.855290    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.855256521 +0000 UTC m=+43.238726432 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:19.305432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:19.305398    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5blf5" event={"ID":"0a5b3a7e-65e1-4ef1-ab25-9944805aac4f","Type":"ContainerStarted","Data":"86aeb4b77e2faa5935d39e4c279d1d778797f7d1b94b038123c11ec0614aed1a"}
Jul 16 17:35:19.307922 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:19.307898    2579 generic.go:356] "Generic (PLEG): container finished" podID="813304a6-08e0-48ab-a660-f8acfe893c99" containerID="4c00f0fe3b67a360b8a9390402fd296f05d08efeacba7ddcd3d2baecdec83234" exitCode=0
Jul 16 17:35:19.308037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:19.307933    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerDied","Data":"4c00f0fe3b67a360b8a9390402fd296f05d08efeacba7ddcd3d2baecdec83234"}
Jul 16 17:35:19.328086 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:19.328044    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5blf5" podStartSLOduration=1.310830843 podStartE2EDuration="5.328030442s" totalImagesPullingTime="4.017199599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:19.326795966 +0000 UTC m=+40.710265898" watchObservedRunningTime="2026-07-16 17:35:19.328030442 +0000 UTC m=+40.711500375"
Jul 16 17:35:20.313332 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:20.313297    2579 generic.go:356] "Generic (PLEG): container finished" podID="813304a6-08e0-48ab-a660-f8acfe893c99" containerID="a8ee996f28372305e16cb59a85e9908d79f7269acc571c02fe435193188f0219" exitCode=0
Jul 16 17:35:20.313825 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:20.313391    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerDied","Data":"a8ee996f28372305e16cb59a85e9908d79f7269acc571c02fe435193188f0219"}
Jul 16 17:35:21.318599 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:21.318568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" event={"ID":"813304a6-08e0-48ab-a660-f8acfe893c99","Type":"ContainerStarted","Data":"024685134d6c378a452c70b6a8ca8521f900c5ece802847d55319733095a5945"}
Jul 16 17:35:21.362750 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:21.362705    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pgzsj" podStartSLOduration=9.408853515 podStartE2EDuration="42.362690049s" totalImagesPullingTime="32.953836534s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:21.36224545 +0000 UTC m=+42.745715383" watchObservedRunningTime="2026-07-16 17:35:21.362690049 +0000 UTC m=+42.746159988"
Jul 16 17:35:21.884044 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:21.884008    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:21.884187 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:21.884131    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:21.884238 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:21.884190    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.884173144 +0000 UTC m=+51.267643058 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:29.940077 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:29.940039    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:29.940586 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:29.940152    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:29.940586 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:29.940201    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:45.940188524 +0000 UTC m=+67.323658435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:40.286128 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:40.286100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vm2hw"
Jul 16 17:35:44.837460 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.837421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:44.837460 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.837468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:44.837919 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.837493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:35:44.843473 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.843453    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:35:44.843558 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.843456    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:35:44.843910 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.843893    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:35:44.847972 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:44.847957    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 17:35:44.848068 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:44.848029    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs podName:b72558f3-3a3d-480a-9540-c427f35bfc10 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:48.848012115 +0000 UTC m=+130.231482028 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs") pod "network-metrics-daemon-d5x6f" (UID: "b72558f3-3a3d-480a-9540-c427f35bfc10") : secret "metrics-daemon-secret" not found
Jul 16 17:35:44.850900 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.850881    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:35:44.851195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.851180    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec9208e8-03c0-4bf2-85f6-7e1470d0da5f-original-pull-secret\") pod \"global-pull-secret-syncer-rngnz\" (UID: \"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f\") " pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:44.860569 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:44.860551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbbx4\" (UniqueName: \"kubernetes.io/projected/e544922c-45ee-4046-9bac-4c20b78fb57d-kube-api-access-kbbx4\") pod \"network-check-target-jxtmc\" (UID: \"e544922c-45ee-4046-9bac-4c20b78fb57d\") " pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:45.115540 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.115468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-28hgh\""
Jul 16 17:35:45.122369 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.122351    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:45.126570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.126549    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rngnz"
Jul 16 17:35:45.280633 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.280607    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rngnz"]
Jul 16 17:35:45.282746 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:35:45.282719    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9208e8_03c0_4bf2_85f6_7e1470d0da5f.slice/crio-851d1f49a89ef9f4248388c00e2f4d07c12a8660eaf608f5d584bc6cd82a32c4 WatchSource:0}: Error finding container 851d1f49a89ef9f4248388c00e2f4d07c12a8660eaf608f5d584bc6cd82a32c4: Status 404 returned error can't find the container with id 851d1f49a89ef9f4248388c00e2f4d07c12a8660eaf608f5d584bc6cd82a32c4
Jul 16 17:35:45.293239 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.293219    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxtmc"]
Jul 16 17:35:45.297711 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:35:45.297689    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode544922c_45ee_4046_9bac_4c20b78fb57d.slice/crio-9813b5f208f0e61198fde1d5c64f3dcfac4d2066d4ea974ef896032958c92fea WatchSource:0}: Error finding container 9813b5f208f0e61198fde1d5c64f3dcfac4d2066d4ea974ef896032958c92fea: Status 404 returned error can't find the container with id 9813b5f208f0e61198fde1d5c64f3dcfac4d2066d4ea974ef896032958c92fea
Jul 16 17:35:45.374061 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.373965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rngnz" event={"ID":"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f","Type":"ContainerStarted","Data":"851d1f49a89ef9f4248388c00e2f4d07c12a8660eaf608f5d584bc6cd82a32c4"}
Jul 16 17:35:45.374943 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.374924    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxtmc" event={"ID":"e544922c-45ee-4046-9bac-4c20b78fb57d","Type":"ContainerStarted","Data":"9813b5f208f0e61198fde1d5c64f3dcfac4d2066d4ea974ef896032958c92fea"}
Jul 16 17:35:45.942670 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:45.942635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:35:45.943021 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:45.942776    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:45.943021 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:45.942840    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls podName:ec720fcf-f94b-433d-907b-a361b69fa425 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:17.942823764 +0000 UTC m=+99.326293684 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls") pod "dns-default-qtjkf" (UID: "ec720fcf-f94b-433d-907b-a361b69fa425") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:50.864811 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.864785    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"]
Jul 16 17:35:50.885831 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.885810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:50.892551 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.892531    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 17:35:50.899257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.895936    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-k6ds4\""
Jul 16 17:35:50.899257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.896054    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 17:35:50.899257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.896976    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 17:35:50.899257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.897303    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 17:35:50.899257 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.898507    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"]
Jul 16 17:35:50.975961 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.975929    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7454758546-dtbkw"]
Jul 16 17:35:50.978463 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.978442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff729380-dbc7-4143-863d-f25fdc3730aa-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:50.978521 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.978481    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:50.978564 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.978539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmgnt\" (UniqueName: \"kubernetes.io/projected/ff729380-dbc7-4143-863d-f25fdc3730aa-kube-api-access-gmgnt\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.000007 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:50.999982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7454758546-dtbkw"]
Jul 16 17:35:51.000136 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.000122    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.003489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.003469    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 17:35:51.005503 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.005484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 17:35:51.005605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.005501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 17:35:51.005605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.005510    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 17:35:51.005859 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.005826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-f7z5m\""
Jul 16 17:35:51.005972 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.005882    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 17:35:51.006037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.006012    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 17:35:51.079108 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff729380-dbc7-4143-863d-f25fdc3730aa-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.079108 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079133    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-stats-auth\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmgnt\" (UniqueName: \"kubernetes.io/projected/ff729380-dbc7-4143-863d-f25fdc3730aa-kube-api-access-gmgnt\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079180    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079194    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlt2c\" (UniqueName: \"kubernetes.io/projected/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-kube-api-access-dlt2c\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.079227    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079242    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-default-certificate\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.079351 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.079313    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls podName:ff729380-dbc7-4143-863d-f25fdc3730aa nodeName:}" failed. No retries permitted until 2026-07-16 17:35:51.579285274 +0000 UTC m=+72.962755202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gfs8k" (UID: "ff729380-dbc7-4143-863d-f25fdc3730aa") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:51.079926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.079907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ff729380-dbc7-4143-863d-f25fdc3730aa-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.096608 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.096585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmgnt\" (UniqueName: \"kubernetes.io/projected/ff729380-dbc7-4143-863d-f25fdc3730aa-kube-api-access-gmgnt\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.180346 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.180278    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.180346 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.180311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.180346 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.180339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlt2c\" (UniqueName: \"kubernetes.io/projected/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-kube-api-access-dlt2c\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.180588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.180357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-default-certificate\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.180588 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.180379    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:51.680362079 +0000 UTC m=+73.063831990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:51.180588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.180466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-stats-auth\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.180588 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.180536    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:51.180787 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.180589    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:51.680572515 +0000 UTC m=+73.064042440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : secret "router-metrics-certs-default" not found
Jul 16 17:35:51.182837 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.182808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-default-certificate\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.182926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.182808    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-stats-auth\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.197986 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.197961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlt2c\" (UniqueName: \"kubernetes.io/projected/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-kube-api-access-dlt2c\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.392041 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.392009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxtmc" event={"ID":"e544922c-45ee-4046-9bac-4c20b78fb57d","Type":"ContainerStarted","Data":"fbc15849d807b2f3756074addf997840692b1904e37f3d5fd3f311fd2390bda9"}
Jul 16 17:35:51.392216 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.392084    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:35:51.393569 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.393545    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rngnz" event={"ID":"ec9208e8-03c0-4bf2-85f6-7e1470d0da5f","Type":"ContainerStarted","Data":"2d0676d55189ba0a7109c0ede8ca08bdaa0c7cc3231030889cabf0ca8904fad9"}
Jul 16 17:35:51.461178 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.461093    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rngnz" podStartSLOduration=66.966175006 podStartE2EDuration="1m12.461078628s" totalImagesPullingTime="5.494903622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:51.460876552 +0000 UTC m=+72.844346487" watchObservedRunningTime="2026-07-16 17:35:51.461078628 +0000 UTC m=+72.844548613"
Jul 16 17:35:51.461309 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.461235    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jxtmc" podStartSLOduration=66.99284198 podStartE2EDuration="1m12.461229761s" totalImagesPullingTime="5.468387781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:51.428674791 +0000 UTC m=+72.812144725" watchObservedRunningTime="2026-07-16 17:35:51.461229761 +0000 UTC m=+72.844699672"
Jul 16 17:35:51.583210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.583173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:51.583375 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.583334    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:51.583420 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.583392    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls podName:ff729380-dbc7-4143-863d-f25fdc3730aa nodeName:}" failed. No retries permitted until 2026-07-16 17:35:52.583377432 +0000 UTC m=+73.966847347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gfs8k" (UID: "ff729380-dbc7-4143-863d-f25fdc3730aa") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:51.684082 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.684053    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.684082 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:51.684084    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:51.684237 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.684185    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:51.684237 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.684207    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:52.684191026 +0000 UTC m=+74.067660942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:51.684351 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:51.684295    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:52.684284504 +0000 UTC m=+74.067754418 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : secret "router-metrics-certs-default" not found
Jul 16 17:35:52.592009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:52.591973    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:52.592515 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:52.592132    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:52.592515 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:52.592217    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls podName:ff729380-dbc7-4143-863d-f25fdc3730aa nodeName:}" failed. No retries permitted until 2026-07-16 17:35:54.592196205 +0000 UTC m=+75.975666117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gfs8k" (UID: "ff729380-dbc7-4143-863d-f25fdc3730aa") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:52.692744 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:52.692715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:52.692864 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:52.692745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:52.692864 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:52.692844    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:52.692928 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:52.692888    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:54.692872975 +0000 UTC m=+76.076342886 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:52.692928 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:52.692912    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:54.692906001 +0000 UTC m=+76.076375912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : secret "router-metrics-certs-default" not found
Jul 16 17:35:54.605937 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:54.605900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:54.606430 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:54.606022    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:54.606430 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:54.606100    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls podName:ff729380-dbc7-4143-863d-f25fdc3730aa nodeName:}" failed. No retries permitted until 2026-07-16 17:35:58.606079466 +0000 UTC m=+79.989549393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gfs8k" (UID: "ff729380-dbc7-4143-863d-f25fdc3730aa") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:54.706977 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:54.706923    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:54.706977 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:54.706984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:54.707236 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:54.707138    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:54.707236 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:54.707224    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:58.707197719 +0000 UTC m=+80.090667637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : secret "router-metrics-certs-default" not found
Jul 16 17:35:54.707925 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:54.707893    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:58.707870432 +0000 UTC m=+80.091340345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:58.635591 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:58.635532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:35:58.636063 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:58.635666    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:58.636063 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:58.635719    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls podName:ff729380-dbc7-4143-863d-f25fdc3730aa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.635705518 +0000 UTC m=+88.019175430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gfs8k" (UID: "ff729380-dbc7-4143-863d-f25fdc3730aa") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:35:58.736146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:58.736108    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:58.736236 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:35:58.736152    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:35:58.736315 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:58.736255    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:58.736315 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:58.736295    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.736258454 +0000 UTC m=+88.119728366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:58.736421 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:35:58.736333    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.73632004 +0000 UTC m=+88.119789951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : secret "router-metrics-certs-default" not found
Jul 16 17:36:01.067524 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.067403    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"]
Jul 16 17:36:01.070089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.070073    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.080336 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.080316    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:36:01.083105 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.083088    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 16 17:36:01.083323 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.083304    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-824fn\""
Jul 16 17:36:01.091566 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.091550    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 16 17:36:01.107130 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.107100    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"]
Jul 16 17:36:01.107809 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.107788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"]
Jul 16 17:36:01.110605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.110590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"
Jul 16 17:36:01.114913 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.114874    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-31.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-132-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 16 17:36:01.114913 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.114884    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-132-31.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-132-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 16 17:36:01.115132 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.114919    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"volume-data-source-validator-dockercfg-thc74\" is forbidden: User \"system:node:ip-10-0-132-31.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-cluster-storage-operator\": no relationship found between node 'ip-10-0-132-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-thc74\"" type="*v1.Secret"
Jul 16 17:36:01.141201 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.141161    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"]
Jul 16 17:36:01.150384 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.150355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lqfn\" (UniqueName: \"kubernetes.io/projected/84926886-1769-4fcf-87d7-1273d1725156-kube-api-access-5lqfn\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.150545 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.150395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.150545 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.150444    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wk8h\" (UniqueName: \"kubernetes.io/projected/25a1915f-e815-4264-a775-dd8cf4709da4-kube-api-access-9wk8h\") pod \"volume-data-source-validator-78495597f6-2jz9x\" (UID: \"25a1915f-e815-4264-a775-dd8cf4709da4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"
Jul 16 17:36:01.251416 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.251382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lqfn\" (UniqueName: \"kubernetes.io/projected/84926886-1769-4fcf-87d7-1273d1725156-kube-api-access-5lqfn\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.251416 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.251422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.251690 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.251443    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk8h\" (UniqueName: \"kubernetes.io/projected/25a1915f-e815-4264-a775-dd8cf4709da4-kube-api-access-9wk8h\") pod \"volume-data-source-validator-78495597f6-2jz9x\" (UID: \"25a1915f-e815-4264-a775-dd8cf4709da4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"
Jul 16 17:36:01.251690 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.251552    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:36:01.251690 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.251615    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls podName:84926886-1769-4fcf-87d7-1273d1725156 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:01.751596562 +0000 UTC m=+83.135066474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls") pod "cluster-samples-operator-65db858894-7mng6" (UID: "84926886-1769-4fcf-87d7-1273d1725156") : secret "samples-operator-tls" not found
Jul 16 17:36:01.272888 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.272854    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lqfn\" (UniqueName: \"kubernetes.io/projected/84926886-1769-4fcf-87d7-1273d1725156-kube-api-access-5lqfn\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.756514 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.756479    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:01.756695 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.756612    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:36:01.756695 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:01.756662    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls podName:84926886-1769-4fcf-87d7-1273d1725156 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:02.756649405 +0000 UTC m=+84.140119316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls") pod "cluster-samples-operator-65db858894-7mng6" (UID: "84926886-1769-4fcf-87d7-1273d1725156") : secret "samples-operator-tls" not found
Jul 16 17:36:01.957386 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:01.957357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-thc74\""
Jul 16 17:36:02.268556 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:02.268514    2579 projected.go:291] Couldn't get configMap openshift-cluster-storage-operator/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 16 17:36:02.272688 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:02.272668    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:36:02.278761 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:02.278731    2579 projected.go:196] Error preparing data for projected volume kube-api-access-9wk8h for pod openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x: failed to sync configmap cache: timed out waiting for the condition
Jul 16 17:36:02.278859 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:02.278815    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/25a1915f-e815-4264-a775-dd8cf4709da4-kube-api-access-9wk8h podName:25a1915f-e815-4264-a775-dd8cf4709da4 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:02.778792687 +0000 UTC m=+84.162262620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9wk8h" (UniqueName: "kubernetes.io/projected/25a1915f-e815-4264-a775-dd8cf4709da4-kube-api-access-9wk8h") pod "volume-data-source-validator-78495597f6-2jz9x" (UID: "25a1915f-e815-4264-a775-dd8cf4709da4") : failed to sync configmap cache: timed out waiting for the condition
Jul 16 17:36:02.280617 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:02.280598    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 17:36:02.763412 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:02.763361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:02.763622 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:02.763507    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:36:02.763622 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:02.763582    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls podName:84926886-1769-4fcf-87d7-1273d1725156 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:04.763561127 +0000 UTC m=+86.147031055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls") pod "cluster-samples-operator-65db858894-7mng6" (UID: "84926886-1769-4fcf-87d7-1273d1725156") : secret "samples-operator-tls" not found
Jul 16 17:36:02.864065 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:02.864030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wk8h\" (UniqueName: \"kubernetes.io/projected/25a1915f-e815-4264-a775-dd8cf4709da4-kube-api-access-9wk8h\") pod \"volume-data-source-validator-78495597f6-2jz9x\" (UID: \"25a1915f-e815-4264-a775-dd8cf4709da4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"
Jul 16 17:36:02.866526 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:02.866495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wk8h\" (UniqueName: \"kubernetes.io/projected/25a1915f-e815-4264-a775-dd8cf4709da4-kube-api-access-9wk8h\") pod \"volume-data-source-validator-78495597f6-2jz9x\" (UID: \"25a1915f-e815-4264-a775-dd8cf4709da4\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"
Jul 16 17:36:02.919890 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:02.919849    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"
Jul 16 17:36:03.053727 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:03.053694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x"]
Jul 16 17:36:03.058544 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:03.058514    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a1915f_e815_4264_a775_dd8cf4709da4.slice/crio-c765c934cdf65c47c2359bdeac4603d051e3122f9218fcc60d9dfe23c2c6cfd2 WatchSource:0}: Error finding container c765c934cdf65c47c2359bdeac4603d051e3122f9218fcc60d9dfe23c2c6cfd2: Status 404 returned error can't find the container with id c765c934cdf65c47c2359bdeac4603d051e3122f9218fcc60d9dfe23c2c6cfd2
Jul 16 17:36:03.423780 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:03.423697    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x" event={"ID":"25a1915f-e815-4264-a775-dd8cf4709da4","Type":"ContainerStarted","Data":"c765c934cdf65c47c2359bdeac4603d051e3122f9218fcc60d9dfe23c2c6cfd2"}
Jul 16 17:36:04.778870 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:04.778831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:04.779255 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:04.779000    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:36:04.779255 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:04.779084    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls podName:84926886-1769-4fcf-87d7-1273d1725156 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:08.779066672 +0000 UTC m=+90.162536585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls") pod "cluster-samples-operator-65db858894-7mng6" (UID: "84926886-1769-4fcf-87d7-1273d1725156") : secret "samples-operator-tls" not found
Jul 16 17:36:05.024279 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.024236    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fsbjf"]
Jul 16 17:36:05.027789 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.027765    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.031114 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.031056    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 17:36:05.031254 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.031195    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 17:36:05.031254 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.031245    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 17:36:05.031376 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.031276    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 17:36:05.032591 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.032569    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wnl5b\""
Jul 16 17:36:05.043645 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.043623    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fsbjf"]
Jul 16 17:36:05.081767 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.081733    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-api-access-9mgck\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.081961 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.081786    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.081961 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.081928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2090a0cd-36b7-43d1-87b8-ed79bb02df11-crio-socket\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.082066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.082001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2090a0cd-36b7-43d1-87b8-ed79bb02df11-data-volume\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.082066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.082047    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182409 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182383    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2090a0cd-36b7-43d1-87b8-ed79bb02df11-crio-socket\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182543 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2090a0cd-36b7-43d1-87b8-ed79bb02df11-data-volume\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182543 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182433    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182543 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182501    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2090a0cd-36b7-43d1-87b8-ed79bb02df11-crio-socket\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182543 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182522    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-api-access-9mgck\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182543 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.182509    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.182764 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182552    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.182764 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.182577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:05.682561377 +0000 UTC m=+87.066031288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.182764 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.182621    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.182764 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.182673    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:05.68266204 +0000 UTC m=+87.066131953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.182764 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.182675    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2090a0cd-36b7-43d1-87b8-ed79bb02df11-data-volume\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.198617 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.198590    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mgck\" (UniqueName: \"kubernetes.io/projected/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-api-access-9mgck\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.430202 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.430164    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x" event={"ID":"25a1915f-e815-4264-a775-dd8cf4709da4","Type":"ContainerStarted","Data":"0a7be514f21df80fc4707a1bcc6d29d1322ab9bb62ffee971df59ec944be8f30"}
Jul 16 17:36:05.449546 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.449498    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-2jz9x" podStartSLOduration=2.352974799 podStartE2EDuration="4.449483073s" totalImagesPullingTime="2.096508274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:05.449040896 +0000 UTC m=+86.832510841" watchObservedRunningTime="2026-07-16 17:36:05.449483073 +0000 UTC m=+86.832953007"
Jul 16 17:36:05.686302 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.686184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.686302 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:05.686305    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:05.686503 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.686355    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.686503 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.686386    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:05.686503 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.686426    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.686410632 +0000 UTC m=+88.069880544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:05.686503 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:05.686441    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:06.686433466 +0000 UTC m=+88.069903378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:06.693147 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:06.693107    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:06.693199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:06.693229    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.693259    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.693326    2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.693329    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.693351    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:08.693334224 +0000 UTC m=+90.076804136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.693369    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls podName:ff729380-dbc7-4143-863d-f25fdc3730aa nodeName:}" failed. No retries permitted until 2026-07-16 17:36:22.693359794 +0000 UTC m=+104.076829705 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-gfs8k" (UID: "ff729380-dbc7-4143-863d-f25fdc3730aa") : secret "cluster-monitoring-operator-tls" not found
Jul 16 17:36:06.693571 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.693383    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:08.693377419 +0000 UTC m=+90.076847331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:06.794242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:06.794207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:06.794242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:06.794243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:06.794450 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.794362    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:36:06.794450 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.794392    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:22.794372887 +0000 UTC m=+104.177842799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : configmap references non-existent config key: service-ca.crt
Jul 16 17:36:06.794450 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:06.794422    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs podName:ec5697a2-8c24-47ed-91e6-d3c2e7827bd9 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:22.794413888 +0000 UTC m=+104.177883801 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs") pod "router-default-7454758546-dtbkw" (UID: "ec5697a2-8c24-47ed-91e6-d3c2e7827bd9") : secret "router-metrics-certs-default" not found
Jul 16 17:36:06.807239 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:06.807216    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-254d8_1bcb2707-bb60-4d1d-b477-bbe7cc04c710/dns-node-resolver/0.log"
Jul 16 17:36:08.007766 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:08.007739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wxrdp_71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1/node-ca/0.log"
Jul 16 17:36:08.416145 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:08.416111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5blf5_0a5b3a7e-65e1-4ef1-ab25-9944805aac4f/serve-healthcheck-canary/0.log"
Jul 16 17:36:08.706760 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:08.706661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:08.706760 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:08.706760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:08.706956 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:08.706818    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:08.706956 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:08.706838    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:08.706956 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:08.706890    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:12.706875775 +0000 UTC m=+94.090345686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:08.706956 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:08.706902    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:12.70689582 +0000 UTC m=+94.090365732 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:08.808156 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:08.808112    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:08.808339 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:08.808256    2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 16 17:36:08.808339 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:08.808336    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls podName:84926886-1769-4fcf-87d7-1273d1725156 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:16.808321325 +0000 UTC m=+98.191791237 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls") pod "cluster-samples-operator-65db858894-7mng6" (UID: "84926886-1769-4fcf-87d7-1273d1725156") : secret "samples-operator-tls" not found
Jul 16 17:36:12.740884 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:12.740846    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:12.741347 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:12.740965    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:12.741347 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:12.741003    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:12.741347 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:12.741077    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:36:12.741347 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:12.741090    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:20.741073418 +0000 UTC m=+102.124543335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:12.741347 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:12.741150    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:20.741136452 +0000 UTC m=+102.124606364 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : configmap "kube-rbac-proxy" not found
Jul 16 17:36:16.870121 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:16.870071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:16.872718 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:16.872687    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/84926886-1769-4fcf-87d7-1273d1725156-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-7mng6\" (UID: \"84926886-1769-4fcf-87d7-1273d1725156\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:16.978556 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:16.978512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"
Jul 16 17:36:17.137518 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:17.137424    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6"]
Jul 16 17:36:17.460224 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:17.460130    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6" event={"ID":"84926886-1769-4fcf-87d7-1273d1725156","Type":"ContainerStarted","Data":"1322da673d830aa105b159c1f38eccf1e423451e9b73f038ef8c6900e73cceb1"}
Jul 16 17:36:17.982083 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:17.982040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:36:17.987625 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:17.987596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec720fcf-f94b-433d-907b-a361b69fa425-metrics-tls\") pod \"dns-default-qtjkf\" (UID: \"ec720fcf-f94b-433d-907b-a361b69fa425\") " pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:36:18.034115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:18.034082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t2d8t\""
Jul 16 17:36:18.041015 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:18.040988    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:36:18.176047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:18.176000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qtjkf"]
Jul 16 17:36:18.178520 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:18.178490    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec720fcf_f94b_433d_907b_a361b69fa425.slice/crio-1de02a1e91f1fc728e121b9d5898a98dfe32e146713fce852656d78f836730f8 WatchSource:0}: Error finding container 1de02a1e91f1fc728e121b9d5898a98dfe32e146713fce852656d78f836730f8: Status 404 returned error can't find the container with id 1de02a1e91f1fc728e121b9d5898a98dfe32e146713fce852656d78f836730f8
Jul 16 17:36:18.463909 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:18.463874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtjkf" event={"ID":"ec720fcf-f94b-433d-907b-a361b69fa425","Type":"ContainerStarted","Data":"1de02a1e91f1fc728e121b9d5898a98dfe32e146713fce852656d78f836730f8"}
Jul 16 17:36:19.471849 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:19.471813    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6" event={"ID":"84926886-1769-4fcf-87d7-1273d1725156","Type":"ContainerStarted","Data":"92da0a42f791e1e00e94859a5ae1d2389f609f1a9293b79512c15b5e7c88203f"}
Jul 16 17:36:20.478423 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:20.478379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6" event={"ID":"84926886-1769-4fcf-87d7-1273d1725156","Type":"ContainerStarted","Data":"b092c2a5d17615de74221eed38e08ea3f5580eaca8c73df45db0c66902680708"}
Jul 16 17:36:20.501006 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:20.500949    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-7mng6" podStartSLOduration=17.303924982 podStartE2EDuration="19.500929746s" totalImagesPullingTime="2.197004764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:20.500607718 +0000 UTC m=+101.884077654" watchObservedRunningTime="2026-07-16 17:36:20.500929746 +0000 UTC m=+101.884399683"
Jul 16 17:36:20.806055 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:20.806022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:20.806210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:20.806094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:20.806210 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:20.806176    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:20.806312 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:20.806243    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls podName:2090a0cd-36b7-43d1-87b8-ed79bb02df11 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:36.806227561 +0000 UTC m=+118.189697473 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fsbjf" (UID: "2090a0cd-36b7-43d1-87b8-ed79bb02df11") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:36:20.806558 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:20.806539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2090a0cd-36b7-43d1-87b8-ed79bb02df11-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:21.483435 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:21.483371    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtjkf" event={"ID":"ec720fcf-f94b-433d-907b-a361b69fa425","Type":"ContainerStarted","Data":"980a19b2924c30a0afdaf11c1dad2fd472531a2c3e8329d562bb53059a402031"}
Jul 16 17:36:21.483810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:21.483479    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtjkf" event={"ID":"ec720fcf-f94b-433d-907b-a361b69fa425","Type":"ContainerStarted","Data":"1ae54690dbf40740572199d22f24689835a530565689c17b0264135e27521f56"}
Jul 16 17:36:21.506348 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:21.506299    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qtjkf" podStartSLOduration=64.588589598 podStartE2EDuration="1m7.50628s" totalImagesPullingTime="2.917690402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:21.505567322 +0000 UTC m=+102.889037256" watchObservedRunningTime="2026-07-16 17:36:21.50628 +0000 UTC m=+102.889749926"
Jul 16 17:36:22.398480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.398451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jxtmc"
Jul 16 17:36:22.486472 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.486437    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:36:22.723156 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.723073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:36:22.725691 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.725663    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ff729380-dbc7-4143-863d-f25fdc3730aa-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-gfs8k\" (UID: \"ff729380-dbc7-4143-863d-f25fdc3730aa\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:36:22.823971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.823931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:22.823971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.823974    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:22.824605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.824585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-service-ca-bundle\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:22.826402 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.826378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec5697a2-8c24-47ed-91e6-d3c2e7827bd9-metrics-certs\") pod \"router-default-7454758546-dtbkw\" (UID: \"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9\") " pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:22.994066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:22.993985    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"
Jul 16 17:36:23.109487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:23.109453    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:23.123983 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:23.123954    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k"]
Jul 16 17:36:23.127843 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:23.127816    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff729380_dbc7_4143_863d_f25fdc3730aa.slice/crio-34ba2c88cb6178c13888ceef86011ea2fb339b6920af133c11ce06e4f441b3bb WatchSource:0}: Error finding container 34ba2c88cb6178c13888ceef86011ea2fb339b6920af133c11ce06e4f441b3bb: Status 404 returned error can't find the container with id 34ba2c88cb6178c13888ceef86011ea2fb339b6920af133c11ce06e4f441b3bb
Jul 16 17:36:23.242486 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:23.242451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7454758546-dtbkw"]
Jul 16 17:36:23.245931 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:23.245865    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5697a2_8c24_47ed_91e6_d3c2e7827bd9.slice/crio-32070edf757e64ac801bc17f373fe854e637337f93184e62a581d84b7dfecf36 WatchSource:0}: Error finding container 32070edf757e64ac801bc17f373fe854e637337f93184e62a581d84b7dfecf36: Status 404 returned error can't find the container with id 32070edf757e64ac801bc17f373fe854e637337f93184e62a581d84b7dfecf36
Jul 16 17:36:23.490966 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:23.490929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k" event={"ID":"ff729380-dbc7-4143-863d-f25fdc3730aa","Type":"ContainerStarted","Data":"34ba2c88cb6178c13888ceef86011ea2fb339b6920af133c11ce06e4f441b3bb"}
Jul 16 17:36:23.492417 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:23.492389    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7454758546-dtbkw" event={"ID":"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9","Type":"ContainerStarted","Data":"ce8038a46fbe6a31a4361c809e883eb3361e71a5dca84db05991e4370621c9b7"}
Jul 16 17:36:23.492549 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:23.492431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7454758546-dtbkw" event={"ID":"ec5697a2-8c24-47ed-91e6-d3c2e7827bd9","Type":"ContainerStarted","Data":"32070edf757e64ac801bc17f373fe854e637337f93184e62a581d84b7dfecf36"}
Jul 16 17:36:24.109980 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:24.109939    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:24.112657 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:24.112628    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:24.139126 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:24.139074    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7454758546-dtbkw" podStartSLOduration=34.139057028 podStartE2EDuration="34.139057028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:23.51670155 +0000 UTC m=+104.900171482" watchObservedRunningTime="2026-07-16 17:36:24.139057028 +0000 UTC m=+105.522526963"
Jul 16 17:36:24.496229 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:24.496146    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:24.497870 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:24.497846    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7454758546-dtbkw"
Jul 16 17:36:26.504318 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:26.504210    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k" event={"ID":"ff729380-dbc7-4143-863d-f25fdc3730aa","Type":"ContainerStarted","Data":"7b3c8f41d6024717f3922b072dea5cc99f14fe70328f1f9638a043e67f222a62"}
Jul 16 17:36:26.528566 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:26.528502    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-gfs8k" podStartSLOduration=33.458944214 podStartE2EDuration="36.528482046s" totalImagesPullingTime="3.069537832s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:26.52773387 +0000 UTC m=+107.911203804" watchObservedRunningTime="2026-07-16 17:36:26.528482046 +0000 UTC m=+107.911951981"
Jul 16 17:36:31.256641 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.256609    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"]
Jul 16 17:36:31.259478 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.259460    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.268065 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.265250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c8d56fccd-2mjw2"]
Jul 16 17:36:31.269342 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.269325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.273779 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.273757    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 17:36:31.274128 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.274113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 17:36:31.277221 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.277201    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 17:36:31.278196 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.278177    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"]
Jul 16 17:36:31.279085 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.279065    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 16 17:36:31.279250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.279235    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-wsnnh\""
Jul 16 17:36:31.279327 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.279314    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 17:36:31.280373 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.280357    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 17:36:31.282988 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.282969    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 17:36:31.284004 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.283987    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 17:36:31.301587 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.301556    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8d56fccd-2mjw2"]
Jul 16 17:36:31.371734 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.371699    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"]
Jul 16 17:36:31.374785 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.374763    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:31.379130 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.379107    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-84k7q\""
Jul 16 17:36:31.379300 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.379109    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 17:36:31.380788 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.380718    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"]
Jul 16 17:36:31.383726 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.383705    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"
Jul 16 17:36:31.389932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.389906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-image-registry-private-configuration\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390034 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.389947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-tls\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390034 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.389976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gcq7\" (UniqueName: \"kubernetes.io/projected/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-kube-api-access-5gcq7\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.390034 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390001    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425cab56-f545-4d51-8e6d-4b88415d39f6-ca-trust-extracted\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390034 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-tmp\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.390253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-certificates\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390062    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-bound-sa-token\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h66w9\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-kube-api-access-h66w9\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-trusted-ca\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-installation-pull-secrets\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.390253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.390178    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-klusterlet-config\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.391926 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:31.391902    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"network-diagnostics-dockercfg-bbbz8\" is forbidden: User \"system:node:ip-10-0-132-31.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-network-diagnostics\": no relationship found between node 'ip-10-0-132-31.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bbbz8\"" type="*v1.Secret"
Jul 16 17:36:31.410378 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.410345    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"]
Jul 16 17:36:31.411753 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.411701    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"]
Jul 16 17:36:31.417748 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.417719    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8d56fccd-2mjw2"]
Jul 16 17:36:31.417989 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:31.417962    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[bound-sa-token ca-trust-extracted image-registry-private-configuration installation-pull-secrets kube-api-access-h66w9 registry-certificates registry-tls trusted-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2" podUID="425cab56-f545-4d51-8e6d-4b88415d39f6"
Jul 16 17:36:31.491167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491061    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f847x\" (UniqueName: \"kubernetes.io/projected/cef031d2-a547-4c74-8d68-82714f26f51a-kube-api-access-f847x\") pod \"network-check-source-869dcfcdcd-p8b85\" (UID: \"cef031d2-a547-4c74-8d68-82714f26f51a\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"
Jul 16 17:36:31.491167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-image-registry-private-configuration\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491185    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-tls\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gcq7\" (UniqueName: \"kubernetes.io/projected/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-kube-api-access-5gcq7\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.491461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425cab56-f545-4d51-8e6d-4b88415d39f6-ca-trust-extracted\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-tmp\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.491461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491404    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-certificates\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491462    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a219f952-93d0-4d26-92f1-5714b80d7390-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-84sw4\" (UID: \"a219f952-93d0-4d26-92f1-5714b80d7390\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-bound-sa-token\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h66w9\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-kube-api-access-h66w9\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-trusted-ca\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-installation-pull-secrets\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-klusterlet-config\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.491719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491694    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-tmp\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.492049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.491842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425cab56-f545-4d51-8e6d-4b88415d39f6-ca-trust-extracted\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.492395 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.492366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-certificates\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.492544 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.492529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-trusted-ca\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.493846 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.493819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-image-registry-private-configuration\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.494106 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.494086    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-installation-pull-secrets\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.494175 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.494100    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-tls\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.494360 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.494340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-klusterlet-config\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.512644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.509013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-bound-sa-token\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.512644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.509034    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gcq7\" (UniqueName: \"kubernetes.io/projected/0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5-kube-api-access-5gcq7\") pod \"klusterlet-addon-workmgr-869bb645c8-dbkpt\" (UID: \"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.512644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.512556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h66w9\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-kube-api-access-h66w9\") pod \"image-registry-6c8d56fccd-2mjw2\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") " pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.521224 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.520749    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.533850 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.533802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:31.572726 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.572679    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:31.592422 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.592396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f847x\" (UniqueName: \"kubernetes.io/projected/cef031d2-a547-4c74-8d68-82714f26f51a-kube-api-access-f847x\") pod \"network-check-source-869dcfcdcd-p8b85\" (UID: \"cef031d2-a547-4c74-8d68-82714f26f51a\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"
Jul 16 17:36:31.592588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.592458    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a219f952-93d0-4d26-92f1-5714b80d7390-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-84sw4\" (UID: \"a219f952-93d0-4d26-92f1-5714b80d7390\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:31.595222 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.595192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a219f952-93d0-4d26-92f1-5714b80d7390-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-84sw4\" (UID: \"a219f952-93d0-4d26-92f1-5714b80d7390\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:31.606251 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.606214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f847x\" (UniqueName: \"kubernetes.io/projected/cef031d2-a547-4c74-8d68-82714f26f51a-kube-api-access-f847x\") pod \"network-check-source-869dcfcdcd-p8b85\" (UID: \"cef031d2-a547-4c74-8d68-82714f26f51a\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"
Jul 16 17:36:31.683685 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.683655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:31.692883 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.692851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-bound-sa-token\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.692893    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-image-registry-private-configuration\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.692919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-tls\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.692951    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-installation-pull-secrets\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.692973    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425cab56-f545-4d51-8e6d-4b88415d39f6-ca-trust-extracted\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693294 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.693026    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-certificates\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693294 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.693067    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h66w9\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-kube-api-access-h66w9\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693294 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.693098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-trusted-ca\") pod \"425cab56-f545-4d51-8e6d-4b88415d39f6\" (UID: \"425cab56-f545-4d51-8e6d-4b88415d39f6\") "
Jul 16 17:36:31.693621 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.693591    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/425cab56-f545-4d51-8e6d-4b88415d39f6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:36:31.693723 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.693698    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:36:31.693905 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.693732    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:36:31.695253 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.695227    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:31.695545 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.695525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:36:31.695782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.695758    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:31.695782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.695764    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:36:31.696298 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.696258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-kube-api-access-h66w9" (OuterVolumeSpecName: "kube-api-access-h66w9") pod "425cab56-f545-4d51-8e6d-4b88415d39f6" (UID: "425cab56-f545-4d51-8e6d-4b88415d39f6"). InnerVolumeSpecName "kube-api-access-h66w9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:31.710561 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.710529    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"]
Jul 16 17:36:31.713803 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:31.713774    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee95def_46e8_4ae7_88d2_9f5d6fc3c7c5.slice/crio-1ae37ccd73a66148f7ca63c70803869df7bd4abb5cf8ddea246522b67e0d8830 WatchSource:0}: Error finding container 1ae37ccd73a66148f7ca63c70803869df7bd4abb5cf8ddea246522b67e0d8830: Status 404 returned error can't find the container with id 1ae37ccd73a66148f7ca63c70803869df7bd4abb5cf8ddea246522b67e0d8830
Jul 16 17:36:31.794315 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794286    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h66w9\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-kube-api-access-h66w9\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794315 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794311    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-trusted-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794315 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794321    2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-bound-sa-token\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794519 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794329    2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-image-registry-private-configuration\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794519 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794338    2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-tls\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794519 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794346    2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/425cab56-f545-4d51-8e6d-4b88415d39f6-installation-pull-secrets\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794519 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794355    2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/425cab56-f545-4d51-8e6d-4b88415d39f6-ca-trust-extracted\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.794519 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.794364    2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/425cab56-f545-4d51-8e6d-4b88415d39f6-registry-certificates\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:36:31.820143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:31.820118    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"]
Jul 16 17:36:31.821810 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:31.821788    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda219f952_93d0_4d26_92f1_5714b80d7390.slice/crio-dc404afdb139ec527f48e5087dd4293e6cfc02de359b76f42d1f9cac63d3033f WatchSource:0}: Error finding container dc404afdb139ec527f48e5087dd4293e6cfc02de359b76f42d1f9cac63d3033f: Status 404 returned error can't find the container with id dc404afdb139ec527f48e5087dd4293e6cfc02de359b76f42d1f9cac63d3033f
Jul 16 17:36:32.215673 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.215641    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bbbz8\""
Jul 16 17:36:32.221929 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.221907    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"
Jul 16 17:36:32.347893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.347862    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85"]
Jul 16 17:36:32.351449 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:32.351422    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef031d2_a547_4c74_8d68_82714f26f51a.slice/crio-2b2d3048353a37351989c0985c057ba704e8b6e40c62dba1d141e587a161edcb WatchSource:0}: Error finding container 2b2d3048353a37351989c0985c057ba704e8b6e40c62dba1d141e587a161edcb: Status 404 returned error can't find the container with id 2b2d3048353a37351989c0985c057ba704e8b6e40c62dba1d141e587a161edcb
Jul 16 17:36:32.494466 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.494387    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qtjkf"
Jul 16 17:36:32.525695 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.525658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4" event={"ID":"a219f952-93d0-4d26-92f1-5714b80d7390","Type":"ContainerStarted","Data":"dc404afdb139ec527f48e5087dd4293e6cfc02de359b76f42d1f9cac63d3033f"}
Jul 16 17:36:32.527256 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.527230    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85" event={"ID":"cef031d2-a547-4c74-8d68-82714f26f51a","Type":"ContainerStarted","Data":"9c817dd237aea77f043f13ed94ee5c4169c50577506f67f2bea0209dca50ce82"}
Jul 16 17:36:32.527475 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.527299    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85" event={"ID":"cef031d2-a547-4c74-8d68-82714f26f51a","Type":"ContainerStarted","Data":"2b2d3048353a37351989c0985c057ba704e8b6e40c62dba1d141e587a161edcb"}
Jul 16 17:36:32.528603 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.528577    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8d56fccd-2mjw2"
Jul 16 17:36:32.528603 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.528595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt" event={"ID":"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5","Type":"ContainerStarted","Data":"1ae37ccd73a66148f7ca63c70803869df7bd4abb5cf8ddea246522b67e0d8830"}
Jul 16 17:36:32.557872 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.557797    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-p8b85" podStartSLOduration=1.557775865 podStartE2EDuration="1.557775865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:32.557698937 +0000 UTC m=+113.941168870" watchObservedRunningTime="2026-07-16 17:36:32.557775865 +0000 UTC m=+113.941245800"
Jul 16 17:36:32.613735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.613661    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8d56fccd-2mjw2"]
Jul 16 17:36:32.616053 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:32.616022    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c8d56fccd-2mjw2"]
Jul 16 17:36:33.103840 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:33.103805    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="425cab56-f545-4d51-8e6d-4b88415d39f6" path="/var/lib/kubelet/pods/425cab56-f545-4d51-8e6d-4b88415d39f6/volumes"
Jul 16 17:36:34.975424 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.975334    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58dbd84867-wk4w9"]
Jul 16 17:36:34.978215 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.978193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:34.982162 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.982136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8mwwj\""
Jul 16 17:36:34.982340 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.982138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 17:36:34.983548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.983522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 17:36:34.983548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.983522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 17:36:34.983710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.983522    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 17:36:34.983710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.983609    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 17:36:34.983850 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.983832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 17:36:34.983850 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.983837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 17:36:34.992333 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:34.992305    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58dbd84867-wk4w9"]
Jul 16 17:36:35.124938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.124889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/6be2fead-a5d8-40d6-a368-d2ea4c665d90-kube-api-access-sgl8f\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.125120 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.124991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-oauth-serving-cert\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.125120 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.125033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-serving-cert\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.125120 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.125078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-config\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.125120 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.125103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-service-ca\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.125429 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.125129    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-oauth-config\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.226384 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.226285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-serving-cert\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.226384 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.226349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-config\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.226384 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.226373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-service-ca\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.226679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.226396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-oauth-config\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.226679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.226487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/6be2fead-a5d8-40d6-a368-d2ea4c665d90-kube-api-access-sgl8f\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.226679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.226516    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-oauth-serving-cert\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.227397 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.227367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-oauth-serving-cert\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.227517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.227367    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-service-ca\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.227517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.227434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-config\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.229226 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.229204    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-oauth-config\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.229374 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.229247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-serving-cert\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.238748 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.238718    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/6be2fead-a5d8-40d6-a368-d2ea4c665d90-kube-api-access-sgl8f\") pod \"console-58dbd84867-wk4w9\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") " pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.289532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.289488    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:35.543136 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.543093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4" event={"ID":"a219f952-93d0-4d26-92f1-5714b80d7390","Type":"ContainerStarted","Data":"f5a8eecd6585f7bcd9c5db4fd79a126ddbb9556229fc052fd130e72b92fd5d19"}
Jul 16 17:36:35.543360 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.543326    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:35.549002 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.548972    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4"
Jul 16 17:36:35.563321 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.563253    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-84sw4" podStartSLOduration=1.832024343 podStartE2EDuration="4.563234522s" totalImagesPullingTime="2.731210179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:35.561151026 +0000 UTC m=+116.944620972" watchObservedRunningTime="2026-07-16 17:36:35.563234522 +0000 UTC m=+116.946704455"
Jul 16 17:36:35.902043 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:35.900941    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58dbd84867-wk4w9"]
Jul 16 17:36:35.906865 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:35.906837    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be2fead_a5d8_40d6_a368_d2ea4c665d90.slice/crio-88893e93cff64a6ee4421524da1fbf4586aece00659b5d9e1b1bbec148188874 WatchSource:0}: Error finding container 88893e93cff64a6ee4421524da1fbf4586aece00659b5d9e1b1bbec148188874: Status 404 returned error can't find the container with id 88893e93cff64a6ee4421524da1fbf4586aece00659b5d9e1b1bbec148188874
Jul 16 17:36:36.547750 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.547709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt" event={"ID":"0ee95def-46e8-4ae7-88d2-9f5d6fc3c7c5","Type":"ContainerStarted","Data":"52a681f66e7e67480da6d975e3c588037ab638a35268a8cd8bc98b0512c28706"}
Jul 16 17:36:36.548217 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.547907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:36.549096 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.549059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dbd84867-wk4w9" event={"ID":"6be2fead-a5d8-40d6-a368-d2ea4c665d90","Type":"ContainerStarted","Data":"88893e93cff64a6ee4421524da1fbf4586aece00659b5d9e1b1bbec148188874"}
Jul 16 17:36:36.549854 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.549835    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt"
Jul 16 17:36:36.566441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.566389    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-869bb645c8-dbkpt" podStartSLOduration=1.4658343280000001 podStartE2EDuration="5.566367854s" totalImagesPullingTime="4.100533526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:36.565677748 +0000 UTC m=+117.949147682" watchObservedRunningTime="2026-07-16 17:36:36.566367854 +0000 UTC m=+117.949837789"
Jul 16 17:36:36.840394 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.840306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:36.842945 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:36.842911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2090a0cd-36b7-43d1-87b8-ed79bb02df11-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fsbjf\" (UID: \"2090a0cd-36b7-43d1-87b8-ed79bb02df11\") " pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:37.138897 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.138813    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fsbjf"
Jul 16 17:36:37.158713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.158679    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f96847fcb-28hs8"]
Jul 16 17:36:37.162388 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.162307    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.172362 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.172248    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 17:36:37.173866 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.173843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f96847fcb-28hs8"]
Jul 16 17:36:37.242782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.242745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-config\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.242965 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.242800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-oauth-config\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.242965 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.242825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-serving-cert\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.242965 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.242933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-service-ca\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.243099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.242971    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-trusted-ca-bundle\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.243099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.243004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-oauth-serving-cert\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.243099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.243033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2lln\" (UniqueName: \"kubernetes.io/projected/72b53958-c4af-4b1d-8ac9-d9477f414d38-kube-api-access-k2lln\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.296250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.296212    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fsbjf"]
Jul 16 17:36:37.301944 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:37.301909    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2090a0cd_36b7_43d1_87b8_ed79bb02df11.slice/crio-340422f96d090420c25a6b043aaa83057918bdb7208e31448cdb67e041141dd3 WatchSource:0}: Error finding container 340422f96d090420c25a6b043aaa83057918bdb7208e31448cdb67e041141dd3: Status 404 returned error can't find the container with id 340422f96d090420c25a6b043aaa83057918bdb7208e31448cdb67e041141dd3
Jul 16 17:36:37.343727 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-oauth-config\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.343938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-serving-cert\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.343938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-service-ca\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.343938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343798    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-trusted-ca-bundle\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.343938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343827    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-oauth-serving-cert\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.343938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343856    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2lln\" (UniqueName: \"kubernetes.io/projected/72b53958-c4af-4b1d-8ac9-d9477f414d38-kube-api-access-k2lln\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.343938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.343924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-config\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.344777 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.344737    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-service-ca\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.344899 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.344842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-oauth-serving-cert\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.344963 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.344944    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-config\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.345022 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.344959    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-trusted-ca-bundle\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.346500 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.346476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-oauth-config\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.346612 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.346599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-serving-cert\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.365973 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.365942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2lln\" (UniqueName: \"kubernetes.io/projected/72b53958-c4af-4b1d-8ac9-d9477f414d38-kube-api-access-k2lln\") pod \"console-f96847fcb-28hs8\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") " pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.474786 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.474693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:37.554382 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.554346    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsbjf" event={"ID":"2090a0cd-36b7-43d1-87b8-ed79bb02df11","Type":"ContainerStarted","Data":"340422f96d090420c25a6b043aaa83057918bdb7208e31448cdb67e041141dd3"}
Jul 16 17:36:37.631010 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:37.630982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f96847fcb-28hs8"]
Jul 16 17:36:37.632259 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:37.632234    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b53958_c4af_4b1d_8ac9_d9477f414d38.slice/crio-d9679952707f6af1da4c49f816186312476d324e51a98fc352509561af535f75 WatchSource:0}: Error finding container d9679952707f6af1da4c49f816186312476d324e51a98fc352509561af535f75: Status 404 returned error can't find the container with id d9679952707f6af1da4c49f816186312476d324e51a98fc352509561af535f75
Jul 16 17:36:38.560656 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:38.560602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f96847fcb-28hs8" event={"ID":"72b53958-c4af-4b1d-8ac9-d9477f414d38","Type":"ContainerStarted","Data":"d9679952707f6af1da4c49f816186312476d324e51a98fc352509561af535f75"}
Jul 16 17:36:39.567579 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:39.567241    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dbd84867-wk4w9" event={"ID":"6be2fead-a5d8-40d6-a368-d2ea4c665d90","Type":"ContainerStarted","Data":"a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14"}
Jul 16 17:36:39.569097 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:39.569065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f96847fcb-28hs8" event={"ID":"72b53958-c4af-4b1d-8ac9-d9477f414d38","Type":"ContainerStarted","Data":"01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726"}
Jul 16 17:36:39.638876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:39.638820    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58dbd84867-wk4w9" podStartSLOduration=2.591473535 podStartE2EDuration="5.63880217s" totalImagesPullingTime="3.047328635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:39.601486787 +0000 UTC m=+120.984956721" watchObservedRunningTime="2026-07-16 17:36:39.63880217 +0000 UTC m=+121.022272162"
Jul 16 17:36:39.639295 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:39.639244    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f96847fcb-28hs8" podStartSLOduration=1.316797951 podStartE2EDuration="2.639231611s" totalImagesPullingTime="1.32243366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:39.638783608 +0000 UTC m=+121.022253542" watchObservedRunningTime="2026-07-16 17:36:39.639231611 +0000 UTC m=+121.022701547"
Jul 16 17:36:41.417985 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.417949    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-j6rrx"]
Jul 16 17:36:41.434764 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.434734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.456836 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.456804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 17:36:41.459781 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.459752    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 17:36:41.460040 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.460019    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 17:36:41.460143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.460047    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 17:36:41.460339 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.460319    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qkjzl\""
Jul 16 17:36:41.580646 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.580845 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-textfile\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.580845 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.580845 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580739    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-tls\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.580845 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580796    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-wtmp\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.580845 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-sys\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.581099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-root\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.581099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-metrics-client-ca\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.581099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.580941    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6w9m\" (UniqueName: \"kubernetes.io/projected/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-kube-api-access-w6w9m\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-tls\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-wtmp\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-sys\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-wtmp\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:41.681741    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-sys\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-root\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681805    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-root\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:36:41.681816    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-tls podName:d9dbc514-88f7-4953-bd27-f3c04cd4d3c0 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:42.181793677 +0000 UTC m=+123.565263592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-tls") pod "node-exporter-j6rrx" (UID: "d9dbc514-88f7-4953-bd27-f3c04cd4d3c0") : secret "node-exporter-tls" not found
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681839    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-metrics-client-ca\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.681904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681884    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6w9m\" (UniqueName: \"kubernetes.io/projected/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-kube-api-access-w6w9m\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.682352 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.681971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.682352 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.682014    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-textfile\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.682352 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.682126    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.682352 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.682340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-textfile\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.682544 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.682505    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-metrics-client-ca\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.682679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.682632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-accelerators-collector-config\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.685142 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.685119    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:41.699102 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:41.699068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6w9m\" (UniqueName: \"kubernetes.io/projected/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-kube-api-access-w6w9m\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:42.186724 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.186681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-tls\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:42.189197 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.189165    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d9dbc514-88f7-4953-bd27-f3c04cd4d3c0-node-exporter-tls\") pod \"node-exporter-j6rrx\" (UID: \"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0\") " pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:42.345492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.345456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-j6rrx"
Jul 16 17:36:42.551246 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.551213    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:36:42.561036 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.560999    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.565208 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565174    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 17:36:42.565427 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565404    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 17:36:42.565502 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565463    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 17:36:42.565502 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 17:36:42.565613 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565550    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 17:36:42.565702 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 17:36:42.565816 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565760    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 17:36:42.565816 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565808    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7z2qb\""
Jul 16 17:36:42.565915 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 17:36:42.565915 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.565902    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 17:36:42.573831 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.573773    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:36:42.691436 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691400    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691436 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691443    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-config-out\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691682 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691469    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691682 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691682 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk2g8\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-kube-api-access-fk2g8\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691682 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691653    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691682 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-web-config\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691763    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-config-volume\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691842    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.691926 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.691862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792578    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-config-out\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792685    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792706    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk2g8\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-kube-api-access-fk2g8\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792760    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-web-config\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.792869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-config-volume\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.793193 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.793193 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.793193 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.793193 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.792963    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.793641 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.793415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.794295 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.794109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.795320 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.795291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.796334 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.796311    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.796610 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.796582    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.797069 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.797013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.797159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.797107    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-config-out\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.797217 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.797156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-config-volume\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.797662 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.797639    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.797785 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.797766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.797843 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.797821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.798820 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.798800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-web-config\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.804469 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.804406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk2g8\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-kube-api-access-fk2g8\") pod \"alertmanager-main-0\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:42.874478 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:42.874444    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:36:43.458585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.458546    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"]
Jul 16 17:36:43.462889 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.462858    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.466749 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.466723    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bd8amu6400l7h\""
Jul 16 17:36:43.466920 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.466754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 17:36:43.467066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.467048    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 17:36:43.467174 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.467155    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 17:36:43.467761 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.467685    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tj87d\""
Jul 16 17:36:43.467761 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.467738    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 17:36:43.467932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.467739    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 17:36:43.497553 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.496430    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"]
Jul 16 17:36:43.514631 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:43.514565    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9dbc514_88f7_4953_bd27_f3c04cd4d3c0.slice/crio-fb6bafa130535c389021b871b31984f2eab0af643ff9faa0d3c032e94b3e61d0 WatchSource:0}: Error finding container fb6bafa130535c389021b871b31984f2eab0af643ff9faa0d3c032e94b3e61d0: Status 404 returned error can't find the container with id fb6bafa130535c389021b871b31984f2eab0af643ff9faa0d3c032e94b3e61d0
Jul 16 17:36:43.587667 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.587597    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6rrx" event={"ID":"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0","Type":"ContainerStarted","Data":"fb6bafa130535c389021b871b31984f2eab0af643ff9faa0d3c032e94b3e61d0"}
Jul 16 17:36:43.599305 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599305 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a330282b-6e6b-4dfa-8b04-f60dcae027e2-metrics-client-ca\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599313    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599368    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-tls\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7xn\" (UniqueName: \"kubernetes.io/projected/a330282b-6e6b-4dfa-8b04-f60dcae027e2-kube-api-access-4l7xn\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599451    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599477    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.599548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.599517    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-grpc-tls\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.681415 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.681388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:36:43.683668 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:43.683637    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3212e110_59fa_4d55_817a_8aa1ba833719.slice/crio-7aec343fe45fa09c35b344d486ab62679350a37b294d712c6e29bd887c7bee16 WatchSource:0}: Error finding container 7aec343fe45fa09c35b344d486ab62679350a37b294d712c6e29bd887c7bee16: Status 404 returned error can't find the container with id 7aec343fe45fa09c35b344d486ab62679350a37b294d712c6e29bd887c7bee16
Jul 16 17:36:43.700790 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.700939 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700801    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a330282b-6e6b-4dfa-8b04-f60dcae027e2-metrics-client-ca\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.700939 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.700939 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700850    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-tls\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.700939 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700876    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7xn\" (UniqueName: \"kubernetes.io/projected/a330282b-6e6b-4dfa-8b04-f60dcae027e2-kube-api-access-4l7xn\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.700939 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.701478 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.700958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.701478 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.701006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-grpc-tls\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.701734 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.701710    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a330282b-6e6b-4dfa-8b04-f60dcae027e2-metrics-client-ca\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.703653 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.703623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-tls\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.703741 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.703674    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.703805 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.703783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.703853 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.703843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.703889 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.703858    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.704434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.704002    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a330282b-6e6b-4dfa-8b04-f60dcae027e2-secret-grpc-tls\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.720872 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.720809    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7xn\" (UniqueName: \"kubernetes.io/projected/a330282b-6e6b-4dfa-8b04-f60dcae027e2-kube-api-access-4l7xn\") pod \"thanos-querier-7f9d78dc5c-hhzqd\" (UID: \"a330282b-6e6b-4dfa-8b04-f60dcae027e2\") " pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.776379 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.776344    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:43.922552 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:43.922520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"]
Jul 16 17:36:43.925515 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:43.925483    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330282b_6e6b_4dfa_8b04_f60dcae027e2.slice/crio-f79bcdbdaa780fdf62dcd6a6477dc5e941aafc342f3fbb9af5248b03047df832 WatchSource:0}: Error finding container f79bcdbdaa780fdf62dcd6a6477dc5e941aafc342f3fbb9af5248b03047df832: Status 404 returned error can't find the container with id f79bcdbdaa780fdf62dcd6a6477dc5e941aafc342f3fbb9af5248b03047df832
Jul 16 17:36:44.593896 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:44.593771    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsbjf" event={"ID":"2090a0cd-36b7-43d1-87b8-ed79bb02df11","Type":"ContainerStarted","Data":"1d38f3fbe928d35bd69847cbc029e4d0271856d9c8bf788cfdddd54e9c4f33ff"}
Jul 16 17:36:44.595176 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:44.595137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"f79bcdbdaa780fdf62dcd6a6477dc5e941aafc342f3fbb9af5248b03047df832"}
Jul 16 17:36:44.596506 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:44.596465    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"7aec343fe45fa09c35b344d486ab62679350a37b294d712c6e29bd887c7bee16"}
Jul 16 17:36:45.290371 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:45.290333    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:45.290560 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:45.290414    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:45.297709 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:45.297679    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:45.602918 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:45.602815    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403" exitCode=0
Jul 16 17:36:45.602918 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:45.602856    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"}
Jul 16 17:36:45.608214 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:45.608178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:36:46.148820 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.148783    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"]
Jul 16 17:36:46.151873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.151850    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.155516 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.155492    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 17:36:46.155516 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.155512    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-cns4d\""
Jul 16 17:36:46.173059 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.173024    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"]
Jul 16 17:36:46.328956 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.328918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a847717-34da-4618-809f-2d42178cf482-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-pwbl4\" (UID: \"2a847717-34da-4618-809f-2d42178cf482\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.329126 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.329037    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tws\" (UniqueName: \"kubernetes.io/projected/2a847717-34da-4618-809f-2d42178cf482-kube-api-access-h4tws\") pod \"monitoring-plugin-5f68df7f58-pwbl4\" (UID: \"2a847717-34da-4618-809f-2d42178cf482\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.429611 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.429532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a847717-34da-4618-809f-2d42178cf482-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-pwbl4\" (UID: \"2a847717-34da-4618-809f-2d42178cf482\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.429611 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.429587    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tws\" (UniqueName: \"kubernetes.io/projected/2a847717-34da-4618-809f-2d42178cf482-kube-api-access-h4tws\") pod \"monitoring-plugin-5f68df7f58-pwbl4\" (UID: \"2a847717-34da-4618-809f-2d42178cf482\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.432452 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.432418    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2a847717-34da-4618-809f-2d42178cf482-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-pwbl4\" (UID: \"2a847717-34da-4618-809f-2d42178cf482\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.445508 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.445475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tws\" (UniqueName: \"kubernetes.io/projected/2a847717-34da-4618-809f-2d42178cf482-kube-api-access-h4tws\") pod \"monitoring-plugin-5f68df7f58-pwbl4\" (UID: \"2a847717-34da-4618-809f-2d42178cf482\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:46.461236 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:46.461203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:47.475635 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:47.475603    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:47.476046 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:47.475754    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:47.480615 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:47.480588    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:47.615955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:47.615921    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:36:47.732461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:47.732381    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58dbd84867-wk4w9"]
Jul 16 17:36:48.432655 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:48.432623    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a847717_34da_4618_809f_2d42178cf482.slice/crio-8ce5f37d1517f77801875934d877f76abd820e1aace1f283c3ddc0caf71c5561 WatchSource:0}: Error finding container 8ce5f37d1517f77801875934d877f76abd820e1aace1f283c3ddc0caf71c5561: Status 404 returned error can't find the container with id 8ce5f37d1517f77801875934d877f76abd820e1aace1f283c3ddc0caf71c5561
Jul 16 17:36:48.447828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.447797    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"]
Jul 16 17:36:48.618047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.618009    2579 generic.go:356] "Generic (PLEG): container finished" podID="d9dbc514-88f7-4953-bd27-f3c04cd4d3c0" containerID="fa015ea4341e279dba49b04ba9bdab5ecbc280ca7f3e5b14e37fb9ec55f06c5b" exitCode=0
Jul 16 17:36:48.618535 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.618100    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6rrx" event={"ID":"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0","Type":"ContainerDied","Data":"fa015ea4341e279dba49b04ba9bdab5ecbc280ca7f3e5b14e37fb9ec55f06c5b"}
Jul 16 17:36:48.623736 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.621703    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsbjf" event={"ID":"2090a0cd-36b7-43d1-87b8-ed79bb02df11","Type":"ContainerStarted","Data":"37cf8c29705b002196e58edfa2415cd449cdf1beeb27208199731a67b4ef202a"}
Jul 16 17:36:48.624023 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.623964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4" event={"ID":"2a847717-34da-4618-809f-2d42178cf482","Type":"ContainerStarted","Data":"8ce5f37d1517f77801875934d877f76abd820e1aace1f283c3ddc0caf71c5561"}
Jul 16 17:36:48.628181 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.628136    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"729530f95826a695a75e68d1952ca5ffc30822906e1953bab9feda4311b88674"}
Jul 16 17:36:48.628333 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.628252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"e4d99915d60b90dbb81c1b5bdca3bd5eef3ebd4254495e506033e99c42dc203e"}
Jul 16 17:36:48.628393 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.628323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"a61ad19c80ad42d135f2207506cec3f8ae2c8d1d8e8865bcebfba1bd1ce9dcbf"}
Jul 16 17:36:48.853677 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.853636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:36:48.856533 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:48.856476    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b72558f3-3a3d-480a-9540-c427f35bfc10-metrics-certs\") pod \"network-metrics-daemon-d5x6f\" (UID: \"b72558f3-3a3d-480a-9540-c427f35bfc10\") " pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:36:49.025111 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:49.025074    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d4hnr\""
Jul 16 17:36:49.032414 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:49.032149    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5x6f"
Jul 16 17:36:49.292770 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:49.292734    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d5x6f"]
Jul 16 17:36:49.636517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:49.636237    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6rrx" event={"ID":"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0","Type":"ContainerStarted","Data":"937c04612730644a97eda3b93d750bb06bc1ae48d3c9a759ca856512e7b0f4fb"}
Jul 16 17:36:49.636517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:49.636322    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-j6rrx" event={"ID":"d9dbc514-88f7-4953-bd27-f3c04cd4d3c0","Type":"ContainerStarted","Data":"e2354305b4911282ced34901272b6f8896854e6224d29f97caa77c7c1518116c"}
Jul 16 17:36:49.639122 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:49.639094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"}
Jul 16 17:36:49.669240 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:36:49.669197    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb72558f3_3a3d_480a_9540_c427f35bfc10.slice/crio-f168ce25cf322f34a29ccf88f31c3224853035ee72278b67f9a861ce0dfa4ad9 WatchSource:0}: Error finding container f168ce25cf322f34a29ccf88f31c3224853035ee72278b67f9a861ce0dfa4ad9: Status 404 returned error can't find the container with id f168ce25cf322f34a29ccf88f31c3224853035ee72278b67f9a861ce0dfa4ad9
Jul 16 17:36:50.647662 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.647617    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d5x6f" event={"ID":"b72558f3-3a3d-480a-9540-c427f35bfc10","Type":"ContainerStarted","Data":"f168ce25cf322f34a29ccf88f31c3224853035ee72278b67f9a861ce0dfa4ad9"}
Jul 16 17:36:50.652129 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.652093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"}
Jul 16 17:36:50.652348 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.652194    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"}
Jul 16 17:36:50.652348 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.652238    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"}
Jul 16 17:36:50.652348 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.652294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"}
Jul 16 17:36:50.654183 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.654119    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4" event={"ID":"2a847717-34da-4618-809f-2d42178cf482","Type":"ContainerStarted","Data":"724276c2365d387997c9977dd51fea0b81c48d4ae96fc39c9ec4295acabdbf9d"}
Jul 16 17:36:50.654589 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.654570    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:50.661140 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.661110    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4"
Jul 16 17:36:50.676319 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.676238    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-j6rrx" podStartSLOduration=4.866715886 podStartE2EDuration="9.676215109s" totalImagesPullingTime="4.721923258s" totalInitContainerRuntime="87.575965ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:49.68002333 +0000 UTC m=+131.063493271" watchObservedRunningTime="2026-07-16 17:36:50.676215109 +0000 UTC m=+132.059685044"
Jul 16 17:36:50.676500 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:50.676427    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-pwbl4" podStartSLOduration=3.392487264 podStartE2EDuration="4.676415994s" totalImagesPullingTime="1.28392873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:50.675335045 +0000 UTC m=+132.058804981" watchObservedRunningTime="2026-07-16 17:36:50.676415994 +0000 UTC m=+132.059885927"
Jul 16 17:36:51.770011 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.769977    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5ffb6f8495-t7vd8"]
Jul 16 17:36:51.787567 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.787533    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.806981 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.806907    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ffb6f8495-t7vd8"]
Jul 16 17:36:51.881711 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.881678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-oauth-config\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.881891 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.881726    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-serving-cert\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.881891 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.881829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-config\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.881891 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.881869    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-trusted-ca-bundle\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.882049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.881955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-oauth-serving-cert\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.882049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.881985    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj755\" (UniqueName: \"kubernetes.io/projected/18b17df1-c7dd-4e88-b9d5-506665acdf39-kube-api-access-wj755\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.882049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.882019    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-service-ca\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.982969 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.982931    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-config\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.982969 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.982968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-trusted-ca-bundle\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.983220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-oauth-serving-cert\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.983220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983019    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj755\" (UniqueName: \"kubernetes.io/projected/18b17df1-c7dd-4e88-b9d5-506665acdf39-kube-api-access-wj755\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.983220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-service-ca\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.983220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983105    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-oauth-config\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.983220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983139    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-serving-cert\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.983896 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-config\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.984028 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.983880    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-oauth-serving-cert\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.984028 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.984016    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-service-ca\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.984650 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.984630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-trusted-ca-bundle\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.986002 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.985977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-serving-cert\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.986415 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.986390    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-oauth-config\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:51.996165 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:51.996135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj755\" (UniqueName: \"kubernetes.io/projected/18b17df1-c7dd-4e88-b9d5-506665acdf39-kube-api-access-wj755\") pod \"console-5ffb6f8495-t7vd8\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") " pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:52.100766 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:52.100503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:36:52.998852 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:52.998749    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5ffb6f8495-t7vd8"]
Jul 16 17:36:53.670308 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.670243    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffb6f8495-t7vd8" event={"ID":"18b17df1-c7dd-4e88-b9d5-506665acdf39","Type":"ContainerStarted","Data":"05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62"}
Jul 16 17:36:53.670479 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.670336    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffb6f8495-t7vd8" event={"ID":"18b17df1-c7dd-4e88-b9d5-506665acdf39","Type":"ContainerStarted","Data":"a87a0fa2a867fcf3a5ef88ad7247b39f8109d35703152f4e985db158e333eeb6"}
Jul 16 17:36:53.673519 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.673486    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerStarted","Data":"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"}
Jul 16 17:36:53.675453 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.675421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fsbjf" event={"ID":"2090a0cd-36b7-43d1-87b8-ed79bb02df11","Type":"ContainerStarted","Data":"e148a875be63b565952fcbe59ce6c3a8eca5901fac50b4b4909eeef5e3f9ae04"}
Jul 16 17:36:53.677048 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.677024    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d5x6f" event={"ID":"b72558f3-3a3d-480a-9540-c427f35bfc10","Type":"ContainerStarted","Data":"246e42acf6fa38d46f76dcc98780e6c8d3aa9ca80f44f153118f8a4400a8c85f"}
Jul 16 17:36:53.677166 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.677062    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d5x6f" event={"ID":"b72558f3-3a3d-480a-9540-c427f35bfc10","Type":"ContainerStarted","Data":"f12a9a36a10ae8da12d06c454a7335dfda2f0981f3ae74d33f6bb7da5aa09433"}
Jul 16 17:36:53.691286 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.691218    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5ffb6f8495-t7vd8" podStartSLOduration=2.691203644 podStartE2EDuration="2.691203644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:53.690030428 +0000 UTC m=+135.073500363" watchObservedRunningTime="2026-07-16 17:36:53.691203644 +0000 UTC m=+135.074673575"
Jul 16 17:36:53.713930 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.713873    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fsbjf" podStartSLOduration=33.347189294 podStartE2EDuration="48.713855528s" totalImagesPullingTime="15.366666234s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:53.711559141 +0000 UTC m=+135.095029075" watchObservedRunningTime="2026-07-16 17:36:53.713855528 +0000 UTC m=+135.097325461"
Jul 16 17:36:53.734206 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.734112    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d5x6f" podStartSLOduration=131.566100412 podStartE2EDuration="2m14.734093818s" totalImagesPullingTime="3.167993406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:53.733468085 +0000 UTC m=+135.116938018" watchObservedRunningTime="2026-07-16 17:36:53.734093818 +0000 UTC m=+135.117563752"
Jul 16 17:36:53.769460 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:53.769402    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.310138138 podStartE2EDuration="11.769383418s" totalImagesPullingTime="7.439485185s" totalInitContainerRuntime="19.760095ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:53.767324092 +0000 UTC m=+135.150794073" watchObservedRunningTime="2026-07-16 17:36:53.769383418 +0000 UTC m=+135.152853352"
Jul 16 17:36:54.684204 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:54.684168    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"2da950977d787cf1bc316f9b364bed3f142dc42d638969d6b136f12b6393f405"}
Jul 16 17:36:55.692256 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:55.692213    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"639382820b84e357c03f83deeeeb62a36e1580cf38a0d935b1c05563c7f53ca3"}
Jul 16 17:36:55.692778 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:55.692300    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" event={"ID":"a330282b-6e6b-4dfa-8b04-f60dcae027e2","Type":"ContainerStarted","Data":"7a72429d9e475a63b8927d3f58d5a1d1f3354c2cf2e3e334e5ae799e7c18f700"}
Jul 16 17:36:55.692778 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:55.692352    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:55.698007 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:55.697981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd"
Jul 16 17:36:55.739525 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:36:55.739465    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7f9d78dc5c-hhzqd" podStartSLOduration=2.352473287 podStartE2EDuration="12.739448312s" totalImagesPullingTime="10.386975025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:55.738194135 +0000 UTC m=+137.121664092" watchObservedRunningTime="2026-07-16 17:36:55.739448312 +0000 UTC m=+137.122918294"
Jul 16 17:37:02.100848 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:02.100782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:37:02.101313 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:02.100870    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:37:02.105931 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:02.105907    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:37:02.730665 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:02.730635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:37:02.792566 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:02.792531    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f96847fcb-28hs8"]
Jul 16 17:37:12.757358 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:12.757308    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58dbd84867-wk4w9" podUID="6be2fead-a5d8-40d6-a368-d2ea4c665d90" containerName="console" containerID="cri-o://a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14" gracePeriod=15
Jul 16 17:37:13.002188 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.002160    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:37:13.069712 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.069687    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-oauth-serving-cert\") pod \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") "
Jul 16 17:37:13.069876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.069748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-serving-cert\") pod \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") "
Jul 16 17:37:13.069876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.069788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-config\") pod \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") "
Jul 16 17:37:13.069876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.069814    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-oauth-config\") pod \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") "
Jul 16 17:37:13.069876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.069851    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-service-ca\") pod \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") "
Jul 16 17:37:13.070041 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.069989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/6be2fead-a5d8-40d6-a368-d2ea4c665d90-kube-api-access-sgl8f\") pod \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\" (UID: \"6be2fead-a5d8-40d6-a368-d2ea4c665d90\") "
Jul 16 17:37:13.070170 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.070140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6be2fead-a5d8-40d6-a368-d2ea4c665d90" (UID: "6be2fead-a5d8-40d6-a368-d2ea4c665d90"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:13.070288 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.070220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-service-ca" (OuterVolumeSpecName: "service-ca") pod "6be2fead-a5d8-40d6-a368-d2ea4c665d90" (UID: "6be2fead-a5d8-40d6-a368-d2ea4c665d90"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:13.070288 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.070225    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-config" (OuterVolumeSpecName: "console-config") pod "6be2fead-a5d8-40d6-a368-d2ea4c665d90" (UID: "6be2fead-a5d8-40d6-a368-d2ea4c665d90"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:13.070398 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.070337    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:13.070398 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.070352    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-service-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:13.070398 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.070361    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6be2fead-a5d8-40d6-a368-d2ea4c665d90-oauth-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:13.072189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.072164    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6be2fead-a5d8-40d6-a368-d2ea4c665d90-kube-api-access-sgl8f" (OuterVolumeSpecName: "kube-api-access-sgl8f") pod "6be2fead-a5d8-40d6-a368-d2ea4c665d90" (UID: "6be2fead-a5d8-40d6-a368-d2ea4c665d90"). InnerVolumeSpecName "kube-api-access-sgl8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:37:13.072389 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.072362    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6be2fead-a5d8-40d6-a368-d2ea4c665d90" (UID: "6be2fead-a5d8-40d6-a368-d2ea4c665d90"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:37:13.072452 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.072408    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6be2fead-a5d8-40d6-a368-d2ea4c665d90" (UID: "6be2fead-a5d8-40d6-a368-d2ea4c665d90"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:37:13.171189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.171148    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-oauth-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:13.171189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.171180    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgl8f\" (UniqueName: \"kubernetes.io/projected/6be2fead-a5d8-40d6-a368-d2ea4c665d90-kube-api-access-sgl8f\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:13.171189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.171190    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6be2fead-a5d8-40d6-a368-d2ea4c665d90-console-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:13.769338 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.769299    2579 generic.go:356] "Generic (PLEG): container finished" podID="6be2fead-a5d8-40d6-a368-d2ea4c665d90" containerID="a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14" exitCode=0
Jul 16 17:37:13.769338 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.769331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58dbd84867-wk4w9"
Jul 16 17:37:13.769893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.769356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dbd84867-wk4w9" event={"ID":"6be2fead-a5d8-40d6-a368-d2ea4c665d90","Type":"ContainerDied","Data":"a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14"}
Jul 16 17:37:13.769893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.769403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58dbd84867-wk4w9" event={"ID":"6be2fead-a5d8-40d6-a368-d2ea4c665d90","Type":"ContainerDied","Data":"88893e93cff64a6ee4421524da1fbf4586aece00659b5d9e1b1bbec148188874"}
Jul 16 17:37:13.769893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.769423    2579 scope.go:117] "RemoveContainer" containerID="a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14"
Jul 16 17:37:13.778948 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.778931    2579 scope.go:117] "RemoveContainer" containerID="a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14"
Jul 16 17:37:13.779289 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:37:13.779250    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14\": container with ID starting with a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14 not found: ID does not exist" containerID="a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14"
Jul 16 17:37:13.779354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.779299    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14"} err="failed to get container status \"a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14\": rpc error: code = NotFound desc = could not find container \"a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14\": container with ID starting with a5e4b154b8e4a93c3a7e032b34808769081452e4f2813abdc30f739c75f34c14 not found: ID does not exist"
Jul 16 17:37:13.793363 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.793333    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58dbd84867-wk4w9"]
Jul 16 17:37:13.798887 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:13.798858    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58dbd84867-wk4w9"]
Jul 16 17:37:15.102489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:15.102453    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6be2fead-a5d8-40d6-a368-d2ea4c665d90" path="/var/lib/kubelet/pods/6be2fead-a5d8-40d6-a368-d2ea4c665d90/volumes"
Jul 16 17:37:27.819081 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:27.819023    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f96847fcb-28hs8" podUID="72b53958-c4af-4b1d-8ac9-d9477f414d38" containerName="console" containerID="cri-o://01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726" gracePeriod=15
Jul 16 17:37:28.061970 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.061941    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:37:28.209579 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209478    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-oauth-config\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.209579 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209530    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-oauth-serving-cert\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.209579 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209569    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-serving-cert\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.209852 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209586    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-trusted-ca-bundle\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.209852 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209630    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-service-ca\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.209852 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-config\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.209852 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.209679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2lln\" (UniqueName: \"kubernetes.io/projected/72b53958-c4af-4b1d-8ac9-d9477f414d38-kube-api-access-k2lln\") pod \"72b53958-c4af-4b1d-8ac9-d9477f414d38\" (UID: \"72b53958-c4af-4b1d-8ac9-d9477f414d38\") "
Jul 16 17:37:28.210081 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.210044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:28.210081 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.210065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-service-ca" (OuterVolumeSpecName: "service-ca") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:28.210181 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.210098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:28.210181 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.210110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-config" (OuterVolumeSpecName: "console-config") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:37:28.211722 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.211696    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:37:28.211868 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.211850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:37:28.211868 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.211859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b53958-c4af-4b1d-8ac9-d9477f414d38-kube-api-access-k2lln" (OuterVolumeSpecName: "kube-api-access-k2lln") pod "72b53958-c4af-4b1d-8ac9-d9477f414d38" (UID: "72b53958-c4af-4b1d-8ac9-d9477f414d38"). InnerVolumeSpecName "kube-api-access-k2lln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:37:28.311278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311239    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-service-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.311278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311287    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.311493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311297    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2lln\" (UniqueName: \"kubernetes.io/projected/72b53958-c4af-4b1d-8ac9-d9477f414d38-kube-api-access-k2lln\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.311493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311307    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-oauth-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.311493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311316    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-oauth-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.311493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311325    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/72b53958-c4af-4b1d-8ac9-d9477f414d38-console-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.311493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.311333    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b53958-c4af-4b1d-8ac9-d9477f414d38-trusted-ca-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:37:28.828684 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.828645    2579 generic.go:356] "Generic (PLEG): container finished" podID="72b53958-c4af-4b1d-8ac9-d9477f414d38" containerID="01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726" exitCode=0
Jul 16 17:37:28.829084 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.828697    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f96847fcb-28hs8"
Jul 16 17:37:28.829084 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.828736    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f96847fcb-28hs8" event={"ID":"72b53958-c4af-4b1d-8ac9-d9477f414d38","Type":"ContainerDied","Data":"01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726"}
Jul 16 17:37:28.829084 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.828795    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f96847fcb-28hs8" event={"ID":"72b53958-c4af-4b1d-8ac9-d9477f414d38","Type":"ContainerDied","Data":"d9679952707f6af1da4c49f816186312476d324e51a98fc352509561af535f75"}
Jul 16 17:37:28.829084 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.828817    2579 scope.go:117] "RemoveContainer" containerID="01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726"
Jul 16 17:37:28.837341 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.837319    2579 scope.go:117] "RemoveContainer" containerID="01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726"
Jul 16 17:37:28.837633 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:37:28.837614    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726\": container with ID starting with 01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726 not found: ID does not exist" containerID="01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726"
Jul 16 17:37:28.837681 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.837641    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726"} err="failed to get container status \"01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726\": rpc error: code = NotFound desc = could not find container \"01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726\": container with ID starting with 01dd573e7421cf3561d8f9683805129c6ff96ac8f6bc6069afe65add89594726 not found: ID does not exist"
Jul 16 17:37:28.858829 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.858798    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f96847fcb-28hs8"]
Jul 16 17:37:28.863965 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:28.863940    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f96847fcb-28hs8"]
Jul 16 17:37:29.102346 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:37:29.102252    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72b53958-c4af-4b1d-8ac9-d9477f414d38" path="/var/lib/kubelet/pods/72b53958-c4af-4b1d-8ac9-d9477f414d38/volumes"
Jul 16 17:38:02.082388 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082349    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:38:02.082959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082639    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="alertmanager" containerID="cri-o://a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42" gracePeriod=120
Jul 16 17:38:02.082959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082697    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-metric" containerID="cri-o://59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97" gracePeriod=120
Jul 16 17:38:02.082959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082761    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-web" containerID="cri-o://1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a" gracePeriod=120
Jul 16 17:38:02.082959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082779    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="config-reloader" containerID="cri-o://fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93" gracePeriod=120
Jul 16 17:38:02.082959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082753    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="prom-label-proxy" containerID="cri-o://41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f" gracePeriod=120
Jul 16 17:38:02.082959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.082799    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy" containerID="cri-o://fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0" gracePeriod=120
Jul 16 17:38:02.969976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.969939    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f" exitCode=0
Jul 16 17:38:02.969976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.969965    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0" exitCode=0
Jul 16 17:38:02.969976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.969974    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93" exitCode=0
Jul 16 17:38:02.969976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.969980    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42" exitCode=0
Jul 16 17:38:02.970242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.970007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"}
Jul 16 17:38:02.970242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.970077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"}
Jul 16 17:38:02.970242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.970108    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"}
Jul 16 17:38:02.970242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:02.970138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"}
Jul 16 17:38:03.324498 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.324474    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:03.408161 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408130    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-web\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408193    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-tls-assets\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408220    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-trusted-ca-bundle\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408241    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-web-config\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408260    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-main-db\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408305    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-main-tls\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-config-volume\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk2g8\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-kube-api-access-fk2g8\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-config-out\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408431    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408456    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-metrics-client-ca\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408484    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-cluster-tls-config\") pod \"3212e110-59fa-4d55-817a-8aa1ba833719\" (UID: \"3212e110-59fa-4d55-817a-8aa1ba833719\") "
Jul 16 17:38:03.408717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408665    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:03.409023 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:38:03.409023 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.408907    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.410038 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.410010    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:03.411673 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.411636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-config-out" (OuterVolumeSpecName: "config-out") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:38:03.412213 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.412129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.412213 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.412176    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.412404 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.412364    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:38:03.412801 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.412774    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-kube-api-access-fk2g8" (OuterVolumeSpecName: "kube-api-access-fk2g8") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "kube-api-access-fk2g8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:38:03.412897 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.412875    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.413078 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.413058    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-config-volume" (OuterVolumeSpecName: "config-volume") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.413949 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.413921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.416218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.416195    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.422589 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.422560    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-web-config" (OuterVolumeSpecName: "web-config") pod "3212e110-59fa-4d55-817a-8aa1ba833719" (UID: "3212e110-59fa-4d55-817a-8aa1ba833719"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510589    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510640    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk2g8\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-kube-api-access-fk2g8\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510656    2579 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-config-out\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510670    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510684    2579 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3212e110-59fa-4d55-817a-8aa1ba833719-metrics-client-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510697    2579 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-cluster-tls-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510710    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.510714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510723    2579 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3212e110-59fa-4d55-817a-8aa1ba833719-tls-assets\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.511089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510738    2579 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-web-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.511089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510750    2579 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3212e110-59fa-4d55-817a-8aa1ba833719-alertmanager-main-db\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.511089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510762    2579 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-secret-alertmanager-main-tls\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.511089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.510773    2579 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3212e110-59fa-4d55-817a-8aa1ba833719-config-volume\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:03.977325 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977289    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97" exitCode=0
Jul 16 17:38:03.977325 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977315    2579 generic.go:356] "Generic (PLEG): container finished" podID="3212e110-59fa-4d55-817a-8aa1ba833719" containerID="1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a" exitCode=0
Jul 16 17:38:03.977588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977392    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"}
Jul 16 17:38:03.977588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977444    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:03.977588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977460    2579 scope.go:117] "RemoveContainer" containerID="41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"
Jul 16 17:38:03.977588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977449    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"}
Jul 16 17:38:03.977790 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.977712    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3212e110-59fa-4d55-817a-8aa1ba833719","Type":"ContainerDied","Data":"7aec343fe45fa09c35b344d486ab62679350a37b294d712c6e29bd887c7bee16"}
Jul 16 17:38:03.985724 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.985559    2579 scope.go:117] "RemoveContainer" containerID="59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"
Jul 16 17:38:03.994089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:03.994069    2579 scope.go:117] "RemoveContainer" containerID="fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"
Jul 16 17:38:04.001273 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.001240    2579 scope.go:117] "RemoveContainer" containerID="1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"
Jul 16 17:38:04.007631 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.007605    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:38:04.009516 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.009497    2579 scope.go:117] "RemoveContainer" containerID="fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"
Jul 16 17:38:04.016778 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.016755    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:38:04.017464 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.017451    2579 scope.go:117] "RemoveContainer" containerID="a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"
Jul 16 17:38:04.025027 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.025004    2579 scope.go:117] "RemoveContainer" containerID="520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"
Jul 16 17:38:04.032388 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.032369    2579 scope.go:117] "RemoveContainer" containerID="41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"
Jul 16 17:38:04.032653 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.032631    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f\": container with ID starting with 41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f not found: ID does not exist" containerID="41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"
Jul 16 17:38:04.032707 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.032659    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"} err="failed to get container status \"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f\": rpc error: code = NotFound desc = could not find container \"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f\": container with ID starting with 41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f not found: ID does not exist"
Jul 16 17:38:04.032707 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.032680    2579 scope.go:117] "RemoveContainer" containerID="59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"
Jul 16 17:38:04.032925 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.032906    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97\": container with ID starting with 59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97 not found: ID does not exist" containerID="59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"
Jul 16 17:38:04.032976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.032931    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"} err="failed to get container status \"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97\": rpc error: code = NotFound desc = could not find container \"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97\": container with ID starting with 59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97 not found: ID does not exist"
Jul 16 17:38:04.032976 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.032948    2579 scope.go:117] "RemoveContainer" containerID="fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"
Jul 16 17:38:04.033197 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.033178    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0\": container with ID starting with fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0 not found: ID does not exist" containerID="fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"
Jul 16 17:38:04.033236 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033199    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"} err="failed to get container status \"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0\": rpc error: code = NotFound desc = could not find container \"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0\": container with ID starting with fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0 not found: ID does not exist"
Jul 16 17:38:04.033236 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033212    2579 scope.go:117] "RemoveContainer" containerID="1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"
Jul 16 17:38:04.033465 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.033448    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a\": container with ID starting with 1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a not found: ID does not exist" containerID="1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"
Jul 16 17:38:04.033501 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033469    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"} err="failed to get container status \"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a\": rpc error: code = NotFound desc = could not find container \"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a\": container with ID starting with 1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a not found: ID does not exist"
Jul 16 17:38:04.033501 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033482    2579 scope.go:117] "RemoveContainer" containerID="fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"
Jul 16 17:38:04.033686 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.033669    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93\": container with ID starting with fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93 not found: ID does not exist" containerID="fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"
Jul 16 17:38:04.033727 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033692    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"} err="failed to get container status \"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93\": rpc error: code = NotFound desc = could not find container \"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93\": container with ID starting with fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93 not found: ID does not exist"
Jul 16 17:38:04.033727 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033704    2579 scope.go:117] "RemoveContainer" containerID="a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"
Jul 16 17:38:04.033896 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.033879    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42\": container with ID starting with a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42 not found: ID does not exist" containerID="a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"
Jul 16 17:38:04.033930 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033899    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"} err="failed to get container status \"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42\": rpc error: code = NotFound desc = could not find container \"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42\": container with ID starting with a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42 not found: ID does not exist"
Jul 16 17:38:04.033930 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.033911    2579 scope.go:117] "RemoveContainer" containerID="520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"
Jul 16 17:38:04.034109 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:04.034093    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403\": container with ID starting with 520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403 not found: ID does not exist" containerID="520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"
Jul 16 17:38:04.034143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034113    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"} err="failed to get container status \"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403\": rpc error: code = NotFound desc = could not find container \"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403\": container with ID starting with 520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403 not found: ID does not exist"
Jul 16 17:38:04.034143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034126    2579 scope.go:117] "RemoveContainer" containerID="41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"
Jul 16 17:38:04.034340 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034322    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f"} err="failed to get container status \"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f\": rpc error: code = NotFound desc = could not find container \"41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f\": container with ID starting with 41bc42257d1b34d3dc3bfcec9babc5d7eb4865945be8f8d4282779a7c7bc946f not found: ID does not exist"
Jul 16 17:38:04.034388 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034341    2579 scope.go:117] "RemoveContainer" containerID="59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"
Jul 16 17:38:04.034554 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034538    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97"} err="failed to get container status \"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97\": rpc error: code = NotFound desc = could not find container \"59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97\": container with ID starting with 59a0b60b628303f16cdf39cd9a3c1c2ea3c23a746d07df056e3053991fe9fd97 not found: ID does not exist"
Jul 16 17:38:04.034593 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034555    2579 scope.go:117] "RemoveContainer" containerID="fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"
Jul 16 17:38:04.034787 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034767    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0"} err="failed to get container status \"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0\": rpc error: code = NotFound desc = could not find container \"fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0\": container with ID starting with fe119afbca23311548ddc708a77359d7d1001d8d95cd616694bce96e302c82c0 not found: ID does not exist"
Jul 16 17:38:04.034846 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034788    2579 scope.go:117] "RemoveContainer" containerID="1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"
Jul 16 17:38:04.035003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.034986    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a"} err="failed to get container status \"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a\": rpc error: code = NotFound desc = could not find container \"1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a\": container with ID starting with 1a420dd5a7d66a919ddf56530d18cb00c621440d435d872e4b016a56e708bc0a not found: ID does not exist"
Jul 16 17:38:04.035055 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.035003    2579 scope.go:117] "RemoveContainer" containerID="fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"
Jul 16 17:38:04.035211 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.035195    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93"} err="failed to get container status \"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93\": rpc error: code = NotFound desc = could not find container \"fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93\": container with ID starting with fac14e4c239d4075c8a21cfebc300cfc051bdfec7c855bae798450a34ed21a93 not found: ID does not exist"
Jul 16 17:38:04.035251 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.035211    2579 scope.go:117] "RemoveContainer" containerID="a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"
Jul 16 17:38:04.035435 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.035418    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42"} err="failed to get container status \"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42\": rpc error: code = NotFound desc = could not find container \"a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42\": container with ID starting with a7dcde647434728aae4af3f24c63ae1abc69fabc0feb6d72ee380c0729aeba42 not found: ID does not exist"
Jul 16 17:38:04.035435 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.035434    2579 scope.go:117] "RemoveContainer" containerID="520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"
Jul 16 17:38:04.035623 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.035607    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403"} err="failed to get container status \"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403\": rpc error: code = NotFound desc = could not find container \"520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403\": container with ID starting with 520ff623cb459e3c12e792a112991015e6e5a597fedc81f6faa30b73f28eb403 not found: ID does not exist"
Jul 16 17:38:04.049672 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049647    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:38:04.049929 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049918    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="alertmanager"
Jul 16 17:38:04.049971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049931    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="alertmanager"
Jul 16 17:38:04.049971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049942    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-web"
Jul 16 17:38:04.049971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049948    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-web"
Jul 16 17:38:04.049971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049955    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="config-reloader"
Jul 16 17:38:04.049971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049961    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="config-reloader"
Jul 16 17:38:04.049971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049970    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="init-config-reloader"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049975    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="init-config-reloader"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049987    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be2fead-a5d8-40d6-a368-d2ea4c665d90" containerName="console"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.049993    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be2fead-a5d8-40d6-a368-d2ea4c665d90" containerName="console"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050000    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050006    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050011    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="prom-label-proxy"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050016    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="prom-label-proxy"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050025    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72b53958-c4af-4b1d-8ac9-d9477f414d38" containerName="console"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050030    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="72b53958-c4af-4b1d-8ac9-d9477f414d38" containerName="console"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050035    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-metric"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050040    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-metric"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050088    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="72b53958-c4af-4b1d-8ac9-d9477f414d38" containerName="console"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050094    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="prom-label-proxy"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050100    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="alertmanager"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050105    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="config-reloader"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050110    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050115    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-web"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050121    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" containerName="kube-rbac-proxy-metric"
Jul 16 17:38:04.050146 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.050127    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be2fead-a5d8-40d6-a368-d2ea4c665d90" containerName="console"
Jul 16 17:38:04.055250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.055231    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.059187 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059159    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 17:38:04.059327 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059207    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-7z2qb\""
Jul 16 17:38:04.059494 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059468    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 17:38:04.059588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059497    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 17:38:04.059588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059540    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 17:38:04.059713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059594    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 17:38:04.059713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059678    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 17:38:04.059713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 17:38:04.059878 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.059768    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 17:38:04.067972 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.067939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 17:38:04.073033 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.073003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:38:04.115701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115667    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115709    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd6b012-1335-4277-9bf6-36cbeab4d496-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd6b012-1335-4277-9bf6-36cbeab4d496-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd6b012-1335-4277-9bf6-36cbeab4d496-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp9rz\" (UniqueName: \"kubernetes.io/projected/5fd6b012-1335-4277-9bf6-36cbeab4d496-kube-api-access-kp9rz\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115862    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd6b012-1335-4277-9bf6-36cbeab4d496-config-out\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115879    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115896    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.115924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.116133 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.116133 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.115973    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd6b012-1335-4277-9bf6-36cbeab4d496-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.116133 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.116029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.116133 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.116078    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-web-config\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.216862 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.216821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd6b012-1335-4277-9bf6-36cbeab4d496-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.216862 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.216866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd6b012-1335-4277-9bf6-36cbeab4d496-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.216887    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp9rz\" (UniqueName: \"kubernetes.io/projected/5fd6b012-1335-4277-9bf6-36cbeab4d496-kube-api-access-kp9rz\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.216914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd6b012-1335-4277-9bf6-36cbeab4d496-config-out\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.216939    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217173 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217063    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217173 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217116    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217173 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd6b012-1335-4277-9bf6-36cbeab4d496-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217353 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-web-config\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217506 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217359    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217506 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217393    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd6b012-1335-4277-9bf6-36cbeab4d496-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.217897 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fd6b012-1335-4277-9bf6-36cbeab4d496-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.218260 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.218236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5fd6b012-1335-4277-9bf6-36cbeab4d496-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.218434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.217325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5fd6b012-1335-4277-9bf6-36cbeab4d496-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220155 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.219980    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5fd6b012-1335-4277-9bf6-36cbeab4d496-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220155 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.219978    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5fd6b012-1335-4277-9bf6-36cbeab4d496-config-out\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220155 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.220106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.220323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.220360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-web-config\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220528 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.220485    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.220836 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.220813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.221088 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.221068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.221938 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.221919    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5fd6b012-1335-4277-9bf6-36cbeab4d496-config-volume\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.227252 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.227232    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp9rz\" (UniqueName: \"kubernetes.io/projected/5fd6b012-1335-4277-9bf6-36cbeab4d496-kube-api-access-kp9rz\") pod \"alertmanager-main-0\" (UID: \"5fd6b012-1335-4277-9bf6-36cbeab4d496\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.365718 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.365681    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 17:38:04.511486 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.511328    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 17:38:04.514019 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:38:04.513995    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd6b012_1335_4277_9bf6_36cbeab4d496.slice/crio-3e532b6cfa4f26c676efff49acc556b3d8d117786237390ed5847cab63db5825 WatchSource:0}: Error finding container 3e532b6cfa4f26c676efff49acc556b3d8d117786237390ed5847cab63db5825: Status 404 returned error can't find the container with id 3e532b6cfa4f26c676efff49acc556b3d8d117786237390ed5847cab63db5825
Jul 16 17:38:04.983035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.983002    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fd6b012-1335-4277-9bf6-36cbeab4d496" containerID="c9d93340a4126ad280db31d7d7cd36364d3933915ef23ddf887d9a0eca27431a" exitCode=0
Jul 16 17:38:04.983243 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.983087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerDied","Data":"c9d93340a4126ad280db31d7d7cd36364d3933915ef23ddf887d9a0eca27431a"}
Jul 16 17:38:04.983243 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:04.983177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"3e532b6cfa4f26c676efff49acc556b3d8d117786237390ed5847cab63db5825"}
Jul 16 17:38:05.101950 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.101917    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3212e110-59fa-4d55-817a-8aa1ba833719" path="/var/lib/kubelet/pods/3212e110-59fa-4d55-817a-8aa1ba833719/volumes"
Jul 16 17:38:05.992840 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.992802    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"2eb2c58a9a3014485de1dbc3523130df80258b6a23ba5e40a3452df49794da88"}
Jul 16 17:38:05.993227 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.992881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"73aff4440e4c0f1e3ea59139f654910e549ff7f4ecd01bbb9ff51d36bed9b55f"}
Jul 16 17:38:05.993227 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.992909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"245484cf149abd37d683ba9348bf72e07454eea2dfe55120e2f6ffb7d39c5914"}
Jul 16 17:38:05.993227 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.992939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"37ec136fe85b866a114c49c744117903b005fe4ba1391b283c0a741083e56497"}
Jul 16 17:38:05.993227 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.992965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"a76f613335475e1924b04576d5405d1b3054f314e6f9aeecd67ad3d3cf4559cf"}
Jul 16 17:38:05.993227 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:05.992991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5fd6b012-1335-4277-9bf6-36cbeab4d496","Type":"ContainerStarted","Data":"04af95b0b00d1356fc8314802bdd299680995b1dd8e83d6d3652794ca4f30146"}
Jul 16 17:38:06.058406 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.058345    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.042800137 podStartE2EDuration="2.058326743s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.526606ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:06.057377939 +0000 UTC m=+207.440847873" watchObservedRunningTime="2026-07-16 17:38:06.058326743 +0000 UTC m=+207.441796683"
Jul 16 17:38:06.163493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.163455    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-575fc68c48-9bwv8"]
Jul 16 17:38:06.166925 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.166900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.169920 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.169888    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 16 17:38:06.170068 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.169928    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9mxfm\""
Jul 16 17:38:06.170068 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.169928    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 16 17:38:06.170068 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.169935    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 16 17:38:06.170068 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.170037    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 16 17:38:06.170320 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.170303    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 16 17:38:06.176492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.176472    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 16 17:38:06.191974 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.191944    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-575fc68c48-9bwv8"]
Jul 16 17:38:06.232802 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2f7c\" (UniqueName: \"kubernetes.io/projected/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-kube-api-access-k2f7c\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.232802 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-serving-certs-ca-bundle\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.233037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.233037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-federate-client-tls\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.233037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-telemeter-client-tls\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.233037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232945    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-secret-telemeter-client\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.233037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.232965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.233217 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.233052    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-metrics-client-ca\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.333929 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.333886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-metrics-client-ca\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.333929 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.333937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2f7c\" (UniqueName: \"kubernetes.io/projected/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-kube-api-access-k2f7c\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.333959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-serving-certs-ca-bundle\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.333979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.333996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-federate-client-tls\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.334062    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-telemeter-client-tls\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.334149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-secret-telemeter-client\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.334180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334786 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.334757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-metrics-client-ca\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334786 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.334784    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-serving-certs-ca-bundle\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.334965 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.334935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.336644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.336621    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-federate-client-tls\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.336782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.336766    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.338753 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.337178    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-secret-telemeter-client\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.338753 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.337225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-telemeter-client-tls\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.349255 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.349224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2f7c\" (UniqueName: \"kubernetes.io/projected/a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0-kube-api-access-k2f7c\") pod \"telemeter-client-575fc68c48-9bwv8\" (UID: \"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0\") " pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.477191 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.477144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8"
Jul 16 17:38:06.632503 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.632477    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-575fc68c48-9bwv8"]
Jul 16 17:38:06.633971 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:38:06.633947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3eacfa3_2c8a_4b9f_8297_0f141a2e7dd0.slice/crio-09accc92bf0957b6c91dc4488461bd9f13fb8a982bc67b7b27447b1be832abc7 WatchSource:0}: Error finding container 09accc92bf0957b6c91dc4488461bd9f13fb8a982bc67b7b27447b1be832abc7: Status 404 returned error can't find the container with id 09accc92bf0957b6c91dc4488461bd9f13fb8a982bc67b7b27447b1be832abc7
Jul 16 17:38:06.998225 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:06.998132    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8" event={"ID":"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0","Type":"ContainerStarted","Data":"09accc92bf0957b6c91dc4488461bd9f13fb8a982bc67b7b27447b1be832abc7"}
Jul 16 17:38:09.010044 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.010004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8" event={"ID":"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0","Type":"ContainerStarted","Data":"c9a70c905abccdf52cd727a515e8f5e77398c43e0685d2878d1d9a80bbce22dd"}
Jul 16 17:38:09.010457 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.010063    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8" event={"ID":"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0","Type":"ContainerStarted","Data":"e5633d2aed10f96493956fd42ef4b84c680c210822c59d35dc204c81d8b80b63"}
Jul 16 17:38:09.010457 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.010082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8" event={"ID":"a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0","Type":"ContainerStarted","Data":"674e04f9fb94cc882aaca3980e67f3ae56a321b53d623085a65a42a0f0f7d135"}
Jul 16 17:38:09.039713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.039655    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-575fc68c48-9bwv8" podStartSLOduration=1.498289999 podStartE2EDuration="3.039638742s" totalImagesPullingTime="1.541348743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:09.039448696 +0000 UTC m=+210.422918631" watchObservedRunningTime="2026-07-16 17:38:09.039638742 +0000 UTC m=+210.423108676"
Jul 16 17:38:09.789651 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.789615    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76d6b44fd8-v4rc7"]
Jul 16 17:38:09.793194 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.793175    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.811331 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.811297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d6b44fd8-v4rc7"]
Jul 16 17:38:09.862022 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.861990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-oauth-config\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.862183 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.862035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-oauth-serving-cert\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.862183 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.862057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-service-ca\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.862183 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.862124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5k7\" (UniqueName: \"kubernetes.io/projected/687e26c1-70c2-434c-981c-d21c52ed9ce2-kube-api-access-bh5k7\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.862364 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.862184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-trusted-ca-bundle\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.862364 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.862205    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-config\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.862364 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.862295    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-serving-cert\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963437 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-serving-cert\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963437 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-oauth-config\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-oauth-serving-cert\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-service-ca\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5k7\" (UniqueName: \"kubernetes.io/projected/687e26c1-70c2-434c-981c-d21c52ed9ce2-kube-api-access-bh5k7\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-trusted-ca-bundle\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.963713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.963630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-config\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.964355 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.964325    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-service-ca\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.964533 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.964506    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-config\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.964720 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.964700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-trusted-ca-bundle\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.964820 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.964797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-oauth-serving-cert\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.966099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.966069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-oauth-config\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.966099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.966094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-serving-cert\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:09.977813 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:09.977781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5k7\" (UniqueName: \"kubernetes.io/projected/687e26c1-70c2-434c-981c-d21c52ed9ce2-kube-api-access-bh5k7\") pod \"console-76d6b44fd8-v4rc7\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") " pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:10.103304 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:10.103249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:10.257090 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:10.257052    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76d6b44fd8-v4rc7"]
Jul 16 17:38:10.261805 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:38:10.261770    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687e26c1_70c2_434c_981c_d21c52ed9ce2.slice/crio-ec0acef29906b239808406079ddb5a6545e1239c56d8609d89e5898e34160c7b WatchSource:0}: Error finding container ec0acef29906b239808406079ddb5a6545e1239c56d8609d89e5898e34160c7b: Status 404 returned error can't find the container with id ec0acef29906b239808406079ddb5a6545e1239c56d8609d89e5898e34160c7b
Jul 16 17:38:11.019930 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:11.019897    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d6b44fd8-v4rc7" event={"ID":"687e26c1-70c2-434c-981c-d21c52ed9ce2","Type":"ContainerStarted","Data":"2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a"}
Jul 16 17:38:11.020107 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:11.019944    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d6b44fd8-v4rc7" event={"ID":"687e26c1-70c2-434c-981c-d21c52ed9ce2","Type":"ContainerStarted","Data":"ec0acef29906b239808406079ddb5a6545e1239c56d8609d89e5898e34160c7b"}
Jul 16 17:38:11.047095 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:11.047037    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76d6b44fd8-v4rc7" podStartSLOduration=2.047020703 podStartE2EDuration="2.047020703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:38:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:38:11.044701588 +0000 UTC m=+212.428171524" watchObservedRunningTime="2026-07-16 17:38:11.047020703 +0000 UTC m=+212.430490636"
Jul 16 17:38:20.104125 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:20.104088    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:20.104125 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:20.104137    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:20.109001 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:20.108979    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:21.063132 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:21.063103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:38:21.132684 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:21.132652    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ffb6f8495-t7vd8"]
Jul 16 17:38:46.155047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.154985    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5ffb6f8495-t7vd8" podUID="18b17df1-c7dd-4e88-b9d5-506665acdf39" containerName="console" containerID="cri-o://05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62" gracePeriod=15
Jul 16 17:38:46.398340 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.398310    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:38:46.485118 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485024    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-oauth-serving-cert\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485118 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485080    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-trusted-ca-bundle\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485118 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485110    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-oauth-config\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485136    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wj755\" (UniqueName: \"kubernetes.io/projected/18b17df1-c7dd-4e88-b9d5-506665acdf39-kube-api-access-wj755\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-serving-cert\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485230    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-service-ca\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-config\") pod \"18b17df1-c7dd-4e88-b9d5-506665acdf39\" (UID: \"18b17df1-c7dd-4e88-b9d5-506665acdf39\") "
Jul 16 17:38:46.485628 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485504    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:46.485687 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:46.485725 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485700    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-service-ca" (OuterVolumeSpecName: "service-ca") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:46.485827 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.485806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-config" (OuterVolumeSpecName: "console-config") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:38:46.487462 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.487426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:46.487462 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.487436    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b17df1-c7dd-4e88-b9d5-506665acdf39-kube-api-access-wj755" (OuterVolumeSpecName: "kube-api-access-wj755") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "kube-api-access-wj755". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:38:46.487607 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.487441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "18b17df1-c7dd-4e88-b9d5-506665acdf39" (UID: "18b17df1-c7dd-4e88-b9d5-506665acdf39"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:38:46.586588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586548    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj755\" (UniqueName: \"kubernetes.io/projected/18b17df1-c7dd-4e88-b9d5-506665acdf39-kube-api-access-wj755\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:46.586588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586579    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:46.586588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586588    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-service-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:46.586588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586597    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:46.586588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586604    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-oauth-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:46.586934 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586612    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18b17df1-c7dd-4e88-b9d5-506665acdf39-trusted-ca-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:46.586934 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:46.586622    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18b17df1-c7dd-4e88-b9d5-506665acdf39-console-oauth-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:38:47.163338 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.163305    2579 generic.go:356] "Generic (PLEG): container finished" podID="18b17df1-c7dd-4e88-b9d5-506665acdf39" containerID="05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62" exitCode=0
Jul 16 17:38:47.163817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.163350    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5ffb6f8495-t7vd8"
Jul 16 17:38:47.163817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.163388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffb6f8495-t7vd8" event={"ID":"18b17df1-c7dd-4e88-b9d5-506665acdf39","Type":"ContainerDied","Data":"05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62"}
Jul 16 17:38:47.163817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.163458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5ffb6f8495-t7vd8" event={"ID":"18b17df1-c7dd-4e88-b9d5-506665acdf39","Type":"ContainerDied","Data":"a87a0fa2a867fcf3a5ef88ad7247b39f8109d35703152f4e985db158e333eeb6"}
Jul 16 17:38:47.163817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.163490    2579 scope.go:117] "RemoveContainer" containerID="05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62"
Jul 16 17:38:47.172606 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.172586    2579 scope.go:117] "RemoveContainer" containerID="05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62"
Jul 16 17:38:47.172862 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:38:47.172842    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62\": container with ID starting with 05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62 not found: ID does not exist" containerID="05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62"
Jul 16 17:38:47.172909 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.172877    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62"} err="failed to get container status \"05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62\": rpc error: code = NotFound desc = could not find container \"05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62\": container with ID starting with 05dc9b1c3dfedcad4b2b524de815e1bec1bda7a82a957cf9b82d696e5e6f5c62 not found: ID does not exist"
Jul 16 17:38:47.187576 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.187549    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5ffb6f8495-t7vd8"]
Jul 16 17:38:47.197466 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:47.197444    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5ffb6f8495-t7vd8"]
Jul 16 17:38:49.102060 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:38:49.102029    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18b17df1-c7dd-4e88-b9d5-506665acdf39" path="/var/lib/kubelet/pods/18b17df1-c7dd-4e88-b9d5-506665acdf39/volumes"
Jul 16 17:39:16.603159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.603122    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5bc48db47c-x2k8n"]
Jul 16 17:39:16.603681 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.603501    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18b17df1-c7dd-4e88-b9d5-506665acdf39" containerName="console"
Jul 16 17:39:16.603681 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.603516    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="18b17df1-c7dd-4e88-b9d5-506665acdf39" containerName="console"
Jul 16 17:39:16.603681 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.603586    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="18b17df1-c7dd-4e88-b9d5-506665acdf39" containerName="console"
Jul 16 17:39:16.607591 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.607571    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.619923 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.619891    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc48db47c-x2k8n"]
Jul 16 17:39:16.640762 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-console-config\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.640910 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9llwl\" (UniqueName: \"kubernetes.io/projected/1d662f1d-114f-4f59-bc86-8631006a338c-kube-api-access-9llwl\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.640910 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-service-ca\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.640910 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640880    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-oauth-config\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.640910 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640906    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-serving-cert\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.641058 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-trusted-ca-bundle\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.641058 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.640949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-oauth-serving-cert\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741677 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741641    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-oauth-config\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-serving-cert\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-trusted-ca-bundle\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-oauth-serving-cert\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-console-config\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9llwl\" (UniqueName: \"kubernetes.io/projected/1d662f1d-114f-4f59-bc86-8631006a338c-kube-api-access-9llwl\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.741873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.741795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-service-ca\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.742619 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.742589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-service-ca\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.742739 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.742620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-console-config\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.742739 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.742630    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-trusted-ca-bundle\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.742935 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.742916    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-oauth-serving-cert\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.744062 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.744031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-oauth-config\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.744207 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.744187    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-serving-cert\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.753300 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.753247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9llwl\" (UniqueName: \"kubernetes.io/projected/1d662f1d-114f-4f59-bc86-8631006a338c-kube-api-access-9llwl\") pod \"console-5bc48db47c-x2k8n\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") " pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:16.917045 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:16.916955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:17.058249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:17.058225    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5bc48db47c-x2k8n"]
Jul 16 17:39:17.060039 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:39:17.060011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d662f1d_114f_4f59_bc86_8631006a338c.slice/crio-a219349f220931f5f55d0f95cfdb252a7ccf8a7e669186c249fcb9fb8791fd52 WatchSource:0}: Error finding container a219349f220931f5f55d0f95cfdb252a7ccf8a7e669186c249fcb9fb8791fd52: Status 404 returned error can't find the container with id a219349f220931f5f55d0f95cfdb252a7ccf8a7e669186c249fcb9fb8791fd52
Jul 16 17:39:17.285100 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:17.285011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc48db47c-x2k8n" event={"ID":"1d662f1d-114f-4f59-bc86-8631006a338c","Type":"ContainerStarted","Data":"051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5"}
Jul 16 17:39:17.285100 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:17.285067    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc48db47c-x2k8n" event={"ID":"1d662f1d-114f-4f59-bc86-8631006a338c","Type":"ContainerStarted","Data":"a219349f220931f5f55d0f95cfdb252a7ccf8a7e669186c249fcb9fb8791fd52"}
Jul 16 17:39:17.311097 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:17.311048    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5bc48db47c-x2k8n" podStartSLOduration=1.311034365 podStartE2EDuration="1.311034365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:39:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:39:17.309710212 +0000 UTC m=+278.693180144" watchObservedRunningTime="2026-07-16 17:39:17.311034365 +0000 UTC m=+278.694504297"
Jul 16 17:39:26.917391 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:26.917347    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:26.917785 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:26.917418    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:26.922223 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:26.922196    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:27.328493 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:27.328464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:39:27.385092 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:27.385056    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d6b44fd8-v4rc7"]
Jul 16 17:39:38.957971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:38.957939    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:39:38.963130 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:38.963108    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 17:39:38.964891 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:38.964872    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:39:52.412767 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.412720    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76d6b44fd8-v4rc7" podUID="687e26c1-70c2-434c-981c-d21c52ed9ce2" containerName="console" containerID="cri-o://2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a" gracePeriod=15
Jul 16 17:39:52.654765 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.654742    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:39:52.731302 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731187    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-trusted-ca-bundle\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731302 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-config\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731505 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731305    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-oauth-serving-cert\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731505 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh5k7\" (UniqueName: \"kubernetes.io/projected/687e26c1-70c2-434c-981c-d21c52ed9ce2-kube-api-access-bh5k7\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731505 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731381    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-serving-cert\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731503    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-oauth-config\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731557    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-service-ca\") pod \"687e26c1-70c2-434c-981c-d21c52ed9ce2\" (UID: \"687e26c1-70c2-434c-981c-d21c52ed9ce2\") "
Jul 16 17:39:52.731893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-config" (OuterVolumeSpecName: "console-config") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:39:52.731893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:39:52.732082 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:39:52.732082 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.731975    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-service-ca" (OuterVolumeSpecName: "service-ca") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:39:52.733556 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.733525    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687e26c1-70c2-434c-981c-d21c52ed9ce2-kube-api-access-bh5k7" (OuterVolumeSpecName: "kube-api-access-bh5k7") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "kube-api-access-bh5k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:39:52.733556 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.733547    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:39:52.733683 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.733530    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "687e26c1-70c2-434c-981c-d21c52ed9ce2" (UID: "687e26c1-70c2-434c-981c-d21c52ed9ce2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:39:52.832856 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832804    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-oauth-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:52.832856 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832850    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-service-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:52.832856 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832859    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-trusted-ca-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:52.832856 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832867    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:52.832856 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832875    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/687e26c1-70c2-434c-981c-d21c52ed9ce2-oauth-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:52.833154 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832883    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5k7\" (UniqueName: \"kubernetes.io/projected/687e26c1-70c2-434c-981c-d21c52ed9ce2-kube-api-access-bh5k7\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:52.833154 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:52.832892    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/687e26c1-70c2-434c-981c-d21c52ed9ce2-console-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:39:53.430201 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.430103    2579 generic.go:356] "Generic (PLEG): container finished" podID="687e26c1-70c2-434c-981c-d21c52ed9ce2" containerID="2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a" exitCode=0
Jul 16 17:39:53.430201 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.430153    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76d6b44fd8-v4rc7"
Jul 16 17:39:53.430711 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.430199    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d6b44fd8-v4rc7" event={"ID":"687e26c1-70c2-434c-981c-d21c52ed9ce2","Type":"ContainerDied","Data":"2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a"}
Jul 16 17:39:53.430711 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.430259    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76d6b44fd8-v4rc7" event={"ID":"687e26c1-70c2-434c-981c-d21c52ed9ce2","Type":"ContainerDied","Data":"ec0acef29906b239808406079ddb5a6545e1239c56d8609d89e5898e34160c7b"}
Jul 16 17:39:53.430711 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.430318    2579 scope.go:117] "RemoveContainer" containerID="2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a"
Jul 16 17:39:53.438869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.438848    2579 scope.go:117] "RemoveContainer" containerID="2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a"
Jul 16 17:39:53.439147 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:39:53.439128    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a\": container with ID starting with 2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a not found: ID does not exist" containerID="2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a"
Jul 16 17:39:53.439210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.439157    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a"} err="failed to get container status \"2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a\": rpc error: code = NotFound desc = could not find container \"2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a\": container with ID starting with 2907473f00a1768770cde71811bb991652e0d360c074abd137da3c3c5fea628a not found: ID does not exist"
Jul 16 17:39:53.450505 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.450475    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76d6b44fd8-v4rc7"]
Jul 16 17:39:53.455963 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:53.455942    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76d6b44fd8-v4rc7"]
Jul 16 17:39:55.102040 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:39:55.102007    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="687e26c1-70c2-434c-981c-d21c52ed9ce2" path="/var/lib/kubelet/pods/687e26c1-70c2-434c-981c-d21c52ed9ce2/volumes"
Jul 16 17:40:48.590649 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.590607    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"]
Jul 16 17:40:48.591043 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.590976    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="687e26c1-70c2-434c-981c-d21c52ed9ce2" containerName="console"
Jul 16 17:40:48.591043 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.590989    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="687e26c1-70c2-434c-981c-d21c52ed9ce2" containerName="console"
Jul 16 17:40:48.591159 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.591048    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="687e26c1-70c2-434c-981c-d21c52ed9ce2" containerName="console"
Jul 16 17:40:48.594464 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.594448    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.597782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.597762    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 17:40:48.598068 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.598053    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 17:40:48.598254 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.598240    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nsxr\""
Jul 16 17:40:48.606594 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.606568    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"]
Jul 16 17:40:48.690981 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.690949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.691178 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.691058    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.691178 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.691113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpbw\" (UniqueName: \"kubernetes.io/projected/4ef7a52f-2d8f-4e95-8285-33252f437e83-kube-api-access-nqpbw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.791726 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.791691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.791917 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.791733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpbw\" (UniqueName: \"kubernetes.io/projected/4ef7a52f-2d8f-4e95-8285-33252f437e83-kube-api-access-nqpbw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.791917 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.791784    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.792164 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.792139    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.792164 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.792155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.801250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.801225    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpbw\" (UniqueName: \"kubernetes.io/projected/4ef7a52f-2d8f-4e95-8285-33252f437e83-kube-api-access-nqpbw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:48.907435 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:48.907329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:40:49.036631 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:49.036599    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"]
Jul 16 17:40:49.039421 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:40:49.039389    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef7a52f_2d8f_4e95_8285_33252f437e83.slice/crio-72e484f556c56cb277a3b6a264adcb6808f2fe5fd1925a3c9b9eacd3d91f5d22 WatchSource:0}: Error finding container 72e484f556c56cb277a3b6a264adcb6808f2fe5fd1925a3c9b9eacd3d91f5d22: Status 404 returned error can't find the container with id 72e484f556c56cb277a3b6a264adcb6808f2fe5fd1925a3c9b9eacd3d91f5d22
Jul 16 17:40:49.041384 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:49.041368    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:40:49.648557 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:49.648518    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9" event={"ID":"4ef7a52f-2d8f-4e95-8285-33252f437e83","Type":"ContainerStarted","Data":"72e484f556c56cb277a3b6a264adcb6808f2fe5fd1925a3c9b9eacd3d91f5d22"}
Jul 16 17:40:55.677103 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:55.677008    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerID="490a5b278cce96ff15c7f059aff6b01e7741646dd504c0b09026ebb37d8ff16d" exitCode=0
Jul 16 17:40:55.677489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:55.677093    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9" event={"ID":"4ef7a52f-2d8f-4e95-8285-33252f437e83","Type":"ContainerDied","Data":"490a5b278cce96ff15c7f059aff6b01e7741646dd504c0b09026ebb37d8ff16d"}
Jul 16 17:40:59.695003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:59.694968    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerID="7daaf3bc782f435cb9d97580ca0bf56c213dc99c4a6fbad0734600297966c45f" exitCode=0
Jul 16 17:40:59.695370 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:40:59.695045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9" event={"ID":"4ef7a52f-2d8f-4e95-8285-33252f437e83","Type":"ContainerDied","Data":"7daaf3bc782f435cb9d97580ca0bf56c213dc99c4a6fbad0734600297966c45f"}
Jul 16 17:41:06.728807 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:06.728767    2579 generic.go:356] "Generic (PLEG): container finished" podID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerID="21fc971af2d4b820b500afd2e10225cd08cb2d4bd3d0ef4558cd376b7cdb9118" exitCode=0
Jul 16 17:41:06.729187 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:06.728844    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9" event={"ID":"4ef7a52f-2d8f-4e95-8285-33252f437e83","Type":"ContainerDied","Data":"21fc971af2d4b820b500afd2e10225cd08cb2d4bd3d0ef4558cd376b7cdb9118"}
Jul 16 17:41:07.860039 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.860014    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:41:07.959703 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.959658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-bundle\") pod \"4ef7a52f-2d8f-4e95-8285-33252f437e83\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") "
Jul 16 17:41:07.959876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.959722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqpbw\" (UniqueName: \"kubernetes.io/projected/4ef7a52f-2d8f-4e95-8285-33252f437e83-kube-api-access-nqpbw\") pod \"4ef7a52f-2d8f-4e95-8285-33252f437e83\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") "
Jul 16 17:41:07.959876 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.959794    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-util\") pod \"4ef7a52f-2d8f-4e95-8285-33252f437e83\" (UID: \"4ef7a52f-2d8f-4e95-8285-33252f437e83\") "
Jul 16 17:41:07.960475 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.960445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-bundle" (OuterVolumeSpecName: "bundle") pod "4ef7a52f-2d8f-4e95-8285-33252f437e83" (UID: "4ef7a52f-2d8f-4e95-8285-33252f437e83"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:41:07.961986 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.961959    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef7a52f-2d8f-4e95-8285-33252f437e83-kube-api-access-nqpbw" (OuterVolumeSpecName: "kube-api-access-nqpbw") pod "4ef7a52f-2d8f-4e95-8285-33252f437e83" (UID: "4ef7a52f-2d8f-4e95-8285-33252f437e83"). InnerVolumeSpecName "kube-api-access-nqpbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:41:07.964079 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:07.964052    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-util" (OuterVolumeSpecName: "util") pod "4ef7a52f-2d8f-4e95-8285-33252f437e83" (UID: "4ef7a52f-2d8f-4e95-8285-33252f437e83"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:41:08.060383 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:08.060354    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:41:08.060383 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:08.060381    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpbw\" (UniqueName: \"kubernetes.io/projected/4ef7a52f-2d8f-4e95-8285-33252f437e83-kube-api-access-nqpbw\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:41:08.060577 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:08.060392    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4ef7a52f-2d8f-4e95-8285-33252f437e83-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:41:08.737612 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:08.737588    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9"
Jul 16 17:41:08.737788 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:08.737582    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5662z9" event={"ID":"4ef7a52f-2d8f-4e95-8285-33252f437e83","Type":"ContainerDied","Data":"72e484f556c56cb277a3b6a264adcb6808f2fe5fd1925a3c9b9eacd3d91f5d22"}
Jul 16 17:41:08.737788 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:08.737708    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72e484f556c56cb277a3b6a264adcb6808f2fe5fd1925a3c9b9eacd3d91f5d22"
Jul 16 17:41:10.908929 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.908896    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"]
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909203    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="util"
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909213    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="util"
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909233    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="pull"
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909239    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="pull"
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909249    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="extract"
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909255    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="extract"
Jul 16 17:41:10.909418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.909343    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef7a52f-2d8f-4e95-8285-33252f437e83" containerName="extract"
Jul 16 17:41:10.937651 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.937614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"]
Jul 16 17:41:10.938037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.937729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:10.940955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.940931    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 17:41:10.941093 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.941027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 17:41:10.941093 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.941045    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 17:41:10.941093 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.941083    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-8pxjl\""
Jul 16 17:41:10.986818 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.986788    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjdm\" (UniqueName: \"kubernetes.io/projected/ec8499e6-3a36-40f9-8a38-ab61617d6838-kube-api-access-hbjdm\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc\" (UID: \"ec8499e6-3a36-40f9-8a38-ab61617d6838\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:10.986974 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:10.986848    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ec8499e6-3a36-40f9-8a38-ab61617d6838-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc\" (UID: \"ec8499e6-3a36-40f9-8a38-ab61617d6838\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:11.087652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.087619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjdm\" (UniqueName: \"kubernetes.io/projected/ec8499e6-3a36-40f9-8a38-ab61617d6838-kube-api-access-hbjdm\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc\" (UID: \"ec8499e6-3a36-40f9-8a38-ab61617d6838\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:11.087843 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.087673    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ec8499e6-3a36-40f9-8a38-ab61617d6838-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc\" (UID: \"ec8499e6-3a36-40f9-8a38-ab61617d6838\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:11.089984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.089955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/ec8499e6-3a36-40f9-8a38-ab61617d6838-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc\" (UID: \"ec8499e6-3a36-40f9-8a38-ab61617d6838\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:11.104621 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.104596    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjdm\" (UniqueName: \"kubernetes.io/projected/ec8499e6-3a36-40f9-8a38-ab61617d6838-kube-api-access-hbjdm\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc\" (UID: \"ec8499e6-3a36-40f9-8a38-ab61617d6838\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:11.248469 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.248380    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:11.405638 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.405576    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"]
Jul 16 17:41:11.408172 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:41:11.408140    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8499e6_3a36_40f9_8a38_ab61617d6838.slice/crio-7c18235a91c26e9fc962d7ae8a0c3d6e6a1a67214184c858070d784e135f8667 WatchSource:0}: Error finding container 7c18235a91c26e9fc962d7ae8a0c3d6e6a1a67214184c858070d784e135f8667: Status 404 returned error can't find the container with id 7c18235a91c26e9fc962d7ae8a0c3d6e6a1a67214184c858070d784e135f8667
Jul 16 17:41:11.749997 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:11.749960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc" event={"ID":"ec8499e6-3a36-40f9-8a38-ab61617d6838","Type":"ContainerStarted","Data":"7c18235a91c26e9fc962d7ae8a0c3d6e6a1a67214184c858070d784e135f8667"}
Jul 16 17:41:15.772052 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.772013    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc" event={"ID":"ec8499e6-3a36-40f9-8a38-ab61617d6838","Type":"ContainerStarted","Data":"e0d3d185ab4c7a95240d28f317fab087ffa9cdffe229ae356a9f5e688802bf01"}
Jul 16 17:41:15.772542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.772169    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:15.804112 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.804058    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc" podStartSLOduration=2.22133987 podStartE2EDuration="5.804042706s" totalImagesPullingTime="3.582702836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:15.801171292 +0000 UTC m=+397.184641249" watchObservedRunningTime="2026-07-16 17:41:15.804042706 +0000 UTC m=+397.187512643"
Jul 16 17:41:15.835761 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.835730    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-zdb27"]
Jul 16 17:41:15.858027 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.857998    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-zdb27"]
Jul 16 17:41:15.858246 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.858101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:15.861054 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.861031    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 16 17:41:15.861496 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.861474    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 16 17:41:15.861585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.861473    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-g4k2w\""
Jul 16 17:41:15.930997 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.930954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhgk\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-kube-api-access-lnhgk\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:15.931166 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.931012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:15.931166 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:15.931050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8c26a160-7943-41f0-920c-956aaccb5ecb-cabundle0\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.031553 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.031460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.031553 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.031506    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8c26a160-7943-41f0-920c-956aaccb5ecb-cabundle0\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.031780 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.031573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhgk\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-kube-api-access-lnhgk\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.031780 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.031648    2579 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 16 17:41:16.031780 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.031674    2579 secret.go:281] references non-existent secret key: ca.crt
Jul 16 17:41:16.031780 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.031687    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 17:41:16.031780 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.031703    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-zdb27: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 16 17:41:16.031780 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.031770    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates podName:8c26a160-7943-41f0-920c-956aaccb5ecb nodeName:}" failed. No retries permitted until 2026-07-16 17:41:16.53175 +0000 UTC m=+397.915219925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates") pod "keda-operator-77f648d694-zdb27" (UID: "8c26a160-7943-41f0-920c-956aaccb5ecb") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 16 17:41:16.032178 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.032160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/8c26a160-7943-41f0-920c-956aaccb5ecb-cabundle0\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.052621 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.052595    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhgk\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-kube-api-access-lnhgk\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.535237 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.535197    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:16.535475 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.535350    2579 secret.go:281] references non-existent secret key: ca.crt
Jul 16 17:41:16.535475 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.535378    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 17:41:16.535475 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.535387    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-zdb27: references non-existent secret key: ca.crt
Jul 16 17:41:16.535475 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:16.535443    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates podName:8c26a160-7943-41f0-920c-956aaccb5ecb nodeName:}" failed. No retries permitted until 2026-07-16 17:41:17.535427096 +0000 UTC m=+398.918897008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates") pod "keda-operator-77f648d694-zdb27" (UID: "8c26a160-7943-41f0-920c-956aaccb5ecb") : references non-existent secret key: ca.crt
Jul 16 17:41:16.832241 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.832211    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"]
Jul 16 17:41:16.834351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.834335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:16.837983 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.837964    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 16 17:41:16.849485 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.849454    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"]
Jul 16 17:41:16.938817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.938787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-certificates\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:16.939001 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.938828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a7b61000-eca0-4139-a456-be18d3da5c6a-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:16.939001 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:16.938847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnlm\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-kube-api-access-5pnlm\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.040769 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.040717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-certificates\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.041085 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.041067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a7b61000-eca0-4139-a456-be18d3da5c6a-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.041198 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.041184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnlm\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-kube-api-access-5pnlm\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.041410 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:17.041189    2579 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 16 17:41:17.041519 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:17.041508    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-cr42k: secret "keda-metrics-apiserver-certs" not found
Jul 16 17:41:17.041616 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.041591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/a7b61000-eca0-4139-a456-be18d3da5c6a-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.041736 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:41:17.041725    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-certificates podName:a7b61000-eca0-4139-a456-be18d3da5c6a nodeName:}" failed. No retries permitted until 2026-07-16 17:41:17.541703229 +0000 UTC m=+398.925173144 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-certificates") pod "keda-metrics-apiserver-55557df8c-cr42k" (UID: "a7b61000-eca0-4139-a456-be18d3da5c6a") : secret "keda-metrics-apiserver-certs" not found
Jul 16 17:41:17.089609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.089540    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnlm\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-kube-api-access-5pnlm\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.546535 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.546501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:17.546698 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.546551    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-certificates\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.549076 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.549042    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8c26a160-7943-41f0-920c-956aaccb5ecb-certificates\") pod \"keda-operator-77f648d694-zdb27\" (UID: \"8c26a160-7943-41f0-920c-956aaccb5ecb\") " pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:17.549193 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.549088    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a7b61000-eca0-4139-a456-be18d3da5c6a-certificates\") pod \"keda-metrics-apiserver-55557df8c-cr42k\" (UID: \"a7b61000-eca0-4139-a456-be18d3da5c6a\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.668972 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.668934    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:17.745337 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.745299    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:17.819719 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:17.819694    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-zdb27"]
Jul 16 17:41:17.821610 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:41:17.821583    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c26a160_7943_41f0_920c_956aaccb5ecb.slice/crio-c4c0755d1bb13868df7987322ff8e45fe3cb7bb36dc648b3c05aef6d0fb37e09 WatchSource:0}: Error finding container c4c0755d1bb13868df7987322ff8e45fe3cb7bb36dc648b3c05aef6d0fb37e09: Status 404 returned error can't find the container with id c4c0755d1bb13868df7987322ff8e45fe3cb7bb36dc648b3c05aef6d0fb37e09
Jul 16 17:41:18.122179 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:18.122152    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"]
Jul 16 17:41:18.124340 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:41:18.124302    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7b61000_eca0_4139_a456_be18d3da5c6a.slice/crio-90af49d4ddbcb75187439ec5e67e97d50a0bffa2068d51340db76816d886ec8c WatchSource:0}: Error finding container 90af49d4ddbcb75187439ec5e67e97d50a0bffa2068d51340db76816d886ec8c: Status 404 returned error can't find the container with id 90af49d4ddbcb75187439ec5e67e97d50a0bffa2068d51340db76816d886ec8c
Jul 16 17:41:18.789580 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:18.789498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-zdb27" event={"ID":"8c26a160-7943-41f0-920c-956aaccb5ecb","Type":"ContainerStarted","Data":"c4c0755d1bb13868df7987322ff8e45fe3cb7bb36dc648b3c05aef6d0fb37e09"}
Jul 16 17:41:18.791027 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:18.790993    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k" event={"ID":"a7b61000-eca0-4139-a456-be18d3da5c6a","Type":"ContainerStarted","Data":"90af49d4ddbcb75187439ec5e67e97d50a0bffa2068d51340db76816d886ec8c"}
Jul 16 17:41:21.807422 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:21.807386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k" event={"ID":"a7b61000-eca0-4139-a456-be18d3da5c6a","Type":"ContainerStarted","Data":"5d59b3e5b365ed1ae4b95abaf5cf559c5574f7a96737d53cb8649b99efdc39c4"}
Jul 16 17:41:21.807887 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:21.807485    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:21.808921 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:21.808895    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-zdb27" event={"ID":"8c26a160-7943-41f0-920c-956aaccb5ecb","Type":"ContainerStarted","Data":"dcc6e83e39c913807ab4a71dfe3cd10aae2753d69ac8194b4cd25230f7931f79"}
Jul 16 17:41:21.809048 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:21.809022    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:41:21.887440 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:21.887394    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k" podStartSLOduration=2.5978461729999998 podStartE2EDuration="5.887379613s" totalImagesPullingTime="3.28953344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:21.853885823 +0000 UTC m=+403.237355756" watchObservedRunningTime="2026-07-16 17:41:21.887379613 +0000 UTC m=+403.270849547"
Jul 16 17:41:32.818428 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:32.818396    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-cr42k"
Jul 16 17:41:32.859806 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:32.859754    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-zdb27" podStartSLOduration=14.263604017 podStartE2EDuration="17.859726208s" totalImagesPullingTime="3.596122191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:21.888007329 +0000 UTC m=+403.271477263" watchObservedRunningTime="2026-07-16 17:41:32.859726208 +0000 UTC m=+414.243196164"
Jul 16 17:41:36.779739 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:36.779707    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-lbmzc"
Jul 16 17:41:42.816143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:41:42.816113    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-zdb27"
Jul 16 17:42:27.236920 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.236888    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-d4c968599-926ss"]
Jul 16 17:42:27.239375 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.239335    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.242515 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.242478    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Jul 16 17:42:27.242722 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.242707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 17:42:27.243884 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.243863    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 17:42:27.244000 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.243896    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-85tts\""
Jul 16 17:42:27.304677 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.304651    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-d4c968599-926ss"]
Jul 16 17:42:27.318020 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.317992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbzz\" (UniqueName: \"kubernetes.io/projected/be199c03-6b94-4519-a3d3-bef344f4115d-kube-api-access-7kbzz\") pod \"llmisvc-controller-manager-d4c968599-926ss\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") " pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.318183 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.318039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be199c03-6b94-4519-a3d3-bef344f4115d-cert\") pod \"llmisvc-controller-manager-d4c968599-926ss\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") " pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.419460 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.419424    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbzz\" (UniqueName: \"kubernetes.io/projected/be199c03-6b94-4519-a3d3-bef344f4115d-kube-api-access-7kbzz\") pod \"llmisvc-controller-manager-d4c968599-926ss\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") " pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.419629 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.419480    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be199c03-6b94-4519-a3d3-bef344f4115d-cert\") pod \"llmisvc-controller-manager-d4c968599-926ss\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") " pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.421894 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.421872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be199c03-6b94-4519-a3d3-bef344f4115d-cert\") pod \"llmisvc-controller-manager-d4c968599-926ss\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") " pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.434813 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.434789    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbzz\" (UniqueName: \"kubernetes.io/projected/be199c03-6b94-4519-a3d3-bef344f4115d-kube-api-access-7kbzz\") pod \"llmisvc-controller-manager-d4c968599-926ss\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") " pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.548734 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.548704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:27.681879 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:27.681854    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-d4c968599-926ss"]
Jul 16 17:42:27.683631 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:42:27.683604    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbe199c03_6b94_4519_a3d3_bef344f4115d.slice/crio-2f3443c9584410964270def6000720ffd09429f8397a0fee1680d02e06c76570 WatchSource:0}: Error finding container 2f3443c9584410964270def6000720ffd09429f8397a0fee1680d02e06c76570: Status 404 returned error can't find the container with id 2f3443c9584410964270def6000720ffd09429f8397a0fee1680d02e06c76570
Jul 16 17:42:28.092940 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:28.092906    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" event={"ID":"be199c03-6b94-4519-a3d3-bef344f4115d","Type":"ContainerStarted","Data":"2f3443c9584410964270def6000720ffd09429f8397a0fee1680d02e06c76570"}
Jul 16 17:42:31.110577 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:31.110539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" event={"ID":"be199c03-6b94-4519-a3d3-bef344f4115d","Type":"ContainerStarted","Data":"d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667"}
Jul 16 17:42:31.110951 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:31.110622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:42:31.133161 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:42:31.133062    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" podStartSLOduration=0.937415375 podStartE2EDuration="4.13304794s" totalImagesPullingTime="3.195632565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:31.131063503 +0000 UTC m=+472.514533438" watchObservedRunningTime="2026-07-16 17:42:31.13304794 +0000 UTC m=+472.516517874"
Jul 16 17:43:02.116943 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:02.116906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 17:43:10.074693 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.074658    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"]
Jul 16 17:43:10.082962 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.082940    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.087061 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.087032    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 17:43:10.088190 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.088167    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 17:43:10.088321 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.088170    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nsxr\""
Jul 16 17:43:10.090739 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.090716    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"]
Jul 16 17:43:10.182201 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.182169    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.182403 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.182261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.182403 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.182305    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9prs\" (UniqueName: \"kubernetes.io/projected/4254c324-28c4-4bf7-b4f8-c65a79d302a4-kube-api-access-w9prs\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.283611 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.283572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.283787 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.283648    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.283787 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.283677    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9prs\" (UniqueName: \"kubernetes.io/projected/4254c324-28c4-4bf7-b4f8-c65a79d302a4-kube-api-access-w9prs\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.283949 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.283930    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.284009 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.283962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.295162 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.295136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9prs\" (UniqueName: \"kubernetes.io/projected/4254c324-28c4-4bf7-b4f8-c65a79d302a4-kube-api-access-w9prs\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.393625 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.393542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:10.532934 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:10.532910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"]
Jul 16 17:43:10.535677 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:43:10.535652    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4254c324_28c4_4bf7_b4f8_c65a79d302a4.slice/crio-7571b2f23527266a076a650bdd7cdf73d03325266a33c0f3e2e1a0573ccfc6d9 WatchSource:0}: Error finding container 7571b2f23527266a076a650bdd7cdf73d03325266a33c0f3e2e1a0573ccfc6d9: Status 404 returned error can't find the container with id 7571b2f23527266a076a650bdd7cdf73d03325266a33c0f3e2e1a0573ccfc6d9
Jul 16 17:43:11.296167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:11.296132    2579 generic.go:356] "Generic (PLEG): container finished" podID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerID="8552f28f75f6341bf1685cf595b6be6dfc4d0d78979a1c5ab29071886f341a9b" exitCode=0
Jul 16 17:43:11.296167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:11.296176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg" event={"ID":"4254c324-28c4-4bf7-b4f8-c65a79d302a4","Type":"ContainerDied","Data":"8552f28f75f6341bf1685cf595b6be6dfc4d0d78979a1c5ab29071886f341a9b"}
Jul 16 17:43:11.296701 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:11.296215    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg" event={"ID":"4254c324-28c4-4bf7-b4f8-c65a79d302a4","Type":"ContainerStarted","Data":"7571b2f23527266a076a650bdd7cdf73d03325266a33c0f3e2e1a0573ccfc6d9"}
Jul 16 17:43:12.453714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.453678    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6466cc997d-hbh2z"]
Jul 16 17:43:12.457184 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.457161    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.469195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.469171    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6466cc997d-hbh2z"]
Jul 16 17:43:12.501320 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501290    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-trusted-ca-bundle\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.501463 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501325    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-console-config\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.501463 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501366    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-service-ca\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.501463 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501448    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-oauth-serving-cert\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.501624 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817965e3-c77c-4ee2-89da-170b65ce8276-console-oauth-config\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.501624 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817965e3-c77c-4ee2-89da-170b65ce8276-console-serving-cert\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.501624 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.501599    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfrt\" (UniqueName: \"kubernetes.io/projected/817965e3-c77c-4ee2-89da-170b65ce8276-kube-api-access-dxfrt\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602630 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602594    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfrt\" (UniqueName: \"kubernetes.io/projected/817965e3-c77c-4ee2-89da-170b65ce8276-kube-api-access-dxfrt\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-trusted-ca-bundle\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602667    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-console-config\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-service-ca\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602712    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-oauth-serving-cert\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817965e3-c77c-4ee2-89da-170b65ce8276-console-oauth-config\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.602810 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.602777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817965e3-c77c-4ee2-89da-170b65ce8276-console-serving-cert\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.603511 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.603484    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-oauth-serving-cert\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.603652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.603617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-service-ca\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.603759 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.603617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-console-config\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.606504 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.603646    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/817965e3-c77c-4ee2-89da-170b65ce8276-trusted-ca-bundle\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.607089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.607061    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/817965e3-c77c-4ee2-89da-170b65ce8276-console-oauth-config\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.611117 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.611094    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/817965e3-c77c-4ee2-89da-170b65ce8276-console-serving-cert\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.613679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.613647    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfrt\" (UniqueName: \"kubernetes.io/projected/817965e3-c77c-4ee2-89da-170b65ce8276-kube-api-access-dxfrt\") pod \"console-6466cc997d-hbh2z\" (UID: \"817965e3-c77c-4ee2-89da-170b65ce8276\") " pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.766302 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.766196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:12.901663 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:12.901627    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6466cc997d-hbh2z"]
Jul 16 17:43:12.904621 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:43:12.904598    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817965e3_c77c_4ee2_89da_170b65ce8276.slice/crio-cc7a74fc0e0bad850be4583b68542a85fb573d7d1bae5c4d10eef01f529e6b42 WatchSource:0}: Error finding container cc7a74fc0e0bad850be4583b68542a85fb573d7d1bae5c4d10eef01f529e6b42: Status 404 returned error can't find the container with id cc7a74fc0e0bad850be4583b68542a85fb573d7d1bae5c4d10eef01f529e6b42
Jul 16 17:43:13.308127 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:13.308088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6466cc997d-hbh2z" event={"ID":"817965e3-c77c-4ee2-89da-170b65ce8276","Type":"ContainerStarted","Data":"12adda24ad627dd3fc3374652fa4d5f23acc259412ddd0a8133bf71b65526e4d"}
Jul 16 17:43:13.308349 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:13.308145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6466cc997d-hbh2z" event={"ID":"817965e3-c77c-4ee2-89da-170b65ce8276","Type":"ContainerStarted","Data":"cc7a74fc0e0bad850be4583b68542a85fb573d7d1bae5c4d10eef01f529e6b42"}
Jul 16 17:43:13.344608 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:13.344558    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6466cc997d-hbh2z" podStartSLOduration=1.344540563 podStartE2EDuration="1.344540563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:13.343875737 +0000 UTC m=+514.727345671" watchObservedRunningTime="2026-07-16 17:43:13.344540563 +0000 UTC m=+514.728010553"
Jul 16 17:43:15.319665 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:15.319630    2579 generic.go:356] "Generic (PLEG): container finished" podID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerID="56bec512a10eb4b0994410fd77aa39f518ef466488b7190a57c3f62619dc0269" exitCode=0
Jul 16 17:43:15.320097 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:15.319710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg" event={"ID":"4254c324-28c4-4bf7-b4f8-c65a79d302a4","Type":"ContainerDied","Data":"56bec512a10eb4b0994410fd77aa39f518ef466488b7190a57c3f62619dc0269"}
Jul 16 17:43:16.326683 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:16.326651    2579 generic.go:356] "Generic (PLEG): container finished" podID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerID="b988c2ce45ba7493120408f3b6ce5765b46a4b055d9cb2011c343be949b153be" exitCode=0
Jul 16 17:43:16.327066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:16.326740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg" event={"ID":"4254c324-28c4-4bf7-b4f8-c65a79d302a4","Type":"ContainerDied","Data":"b988c2ce45ba7493120408f3b6ce5765b46a4b055d9cb2011c343be949b153be"}
Jul 16 17:43:17.469516 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.469492    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:17.551137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.551103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-util\") pod \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") "
Jul 16 17:43:17.551137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.551138    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9prs\" (UniqueName: \"kubernetes.io/projected/4254c324-28c4-4bf7-b4f8-c65a79d302a4-kube-api-access-w9prs\") pod \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") "
Jul 16 17:43:17.551356 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.551197    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-bundle\") pod \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\" (UID: \"4254c324-28c4-4bf7-b4f8-c65a79d302a4\") "
Jul 16 17:43:17.551945 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.551925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-bundle" (OuterVolumeSpecName: "bundle") pod "4254c324-28c4-4bf7-b4f8-c65a79d302a4" (UID: "4254c324-28c4-4bf7-b4f8-c65a79d302a4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:43:17.553310 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.553294    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4254c324-28c4-4bf7-b4f8-c65a79d302a4-kube-api-access-w9prs" (OuterVolumeSpecName: "kube-api-access-w9prs") pod "4254c324-28c4-4bf7-b4f8-c65a79d302a4" (UID: "4254c324-28c4-4bf7-b4f8-c65a79d302a4"). InnerVolumeSpecName "kube-api-access-w9prs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:43:17.556128 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.556105    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-util" (OuterVolumeSpecName: "util") pod "4254c324-28c4-4bf7-b4f8-c65a79d302a4" (UID: "4254c324-28c4-4bf7-b4f8-c65a79d302a4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:43:17.652207 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.652134    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:17.652207 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.652159    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9prs\" (UniqueName: \"kubernetes.io/projected/4254c324-28c4-4bf7-b4f8-c65a79d302a4-kube-api-access-w9prs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:17.652207 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:17.652170    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4254c324-28c4-4bf7-b4f8-c65a79d302a4-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:18.337841 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:18.337803    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg" event={"ID":"4254c324-28c4-4bf7-b4f8-c65a79d302a4","Type":"ContainerDied","Data":"7571b2f23527266a076a650bdd7cdf73d03325266a33c0f3e2e1a0573ccfc6d9"}
Jul 16 17:43:18.337984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:18.337846    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bqpbg"
Jul 16 17:43:18.337984 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:18.337856    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7571b2f23527266a076a650bdd7cdf73d03325266a33c0f3e2e1a0573ccfc6d9"
Jul 16 17:43:22.766462 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:22.766420    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:22.766911 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:22.766502    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:22.771432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:22.771406    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:23.212124 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212071    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"]
Jul 16 17:43:23.212483 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212469    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="pull"
Jul 16 17:43:23.212532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212487    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="pull"
Jul 16 17:43:23.212532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212509    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="extract"
Jul 16 17:43:23.212532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212514    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="extract"
Jul 16 17:43:23.212532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212521    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="util"
Jul 16 17:43:23.212532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212527    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="util"
Jul 16 17:43:23.212678 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.212582    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4254c324-28c4-4bf7-b4f8-c65a79d302a4" containerName="extract"
Jul 16 17:43:23.216611 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.216594    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.219572 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.219546    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 16 17:43:23.219842 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.219826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-8vfgt\""
Jul 16 17:43:23.219899 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.219879    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:43:23.231144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.231114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"]
Jul 16 17:43:23.298364 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.298326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zjsw5\" (UID: \"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.298556 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.298409    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8czc\" (UniqueName: \"kubernetes.io/projected/e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d-kube-api-access-x8czc\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zjsw5\" (UID: \"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.363788 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.363760    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6466cc997d-hbh2z"
Jul 16 17:43:23.399735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.399690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zjsw5\" (UID: \"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.399919 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.399823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8czc\" (UniqueName: \"kubernetes.io/projected/e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d-kube-api-access-x8czc\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zjsw5\" (UID: \"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.400118 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.400097    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zjsw5\" (UID: \"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.418097 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.418066    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8czc\" (UniqueName: \"kubernetes.io/projected/e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d-kube-api-access-x8czc\") pod \"cert-manager-operator-controller-manager-684f79c9b9-zjsw5\" (UID: \"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.429637 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.429605    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc48db47c-x2k8n"]
Jul 16 17:43:23.526282 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.526193    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"
Jul 16 17:43:23.665766 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:23.665741    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5"]
Jul 16 17:43:23.667941 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:43:23.667913    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ca0d38_eca6_4fdf_831f_ebcd0743cd2d.slice/crio-084d0735ce311e451b8f43b22835ec7a5ebcf5732470a25b78d0009152cb93f9 WatchSource:0}: Error finding container 084d0735ce311e451b8f43b22835ec7a5ebcf5732470a25b78d0009152cb93f9: Status 404 returned error can't find the container with id 084d0735ce311e451b8f43b22835ec7a5ebcf5732470a25b78d0009152cb93f9
Jul 16 17:43:24.366455 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:24.366421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5" event={"ID":"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d","Type":"ContainerStarted","Data":"084d0735ce311e451b8f43b22835ec7a5ebcf5732470a25b78d0009152cb93f9"}
Jul 16 17:43:25.372471 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:25.372435    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5" event={"ID":"e1ca0d38-eca6-4fdf-831f-ebcd0743cd2d","Type":"ContainerStarted","Data":"5bd084304370ad7fd7ab3299f237059004f208a716f3e5d4443b52d95aef31ba"}
Jul 16 17:43:25.412118 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:25.412060    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-zjsw5" podStartSLOduration=0.90038602 podStartE2EDuration="2.412038728s" totalImagesPullingTime="1.511652708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:25.405736441 +0000 UTC m=+526.789206376" watchObservedRunningTime="2026-07-16 17:43:25.412038728 +0000 UTC m=+526.795508661"
Jul 16 17:43:26.810351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.810311    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"]
Jul 16 17:43:26.813957 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.813942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:26.817302 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.817281    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 17:43:26.817434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.817281    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nsxr\""
Jul 16 17:43:26.818480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.818467    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 17:43:26.825710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.825690    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"]
Jul 16 17:43:26.930655 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.930621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrl5q\" (UniqueName: \"kubernetes.io/projected/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-kube-api-access-qrl5q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:26.930823 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.930665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:26.930823 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:26.930728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.031408 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.031369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrl5q\" (UniqueName: \"kubernetes.io/projected/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-kube-api-access-qrl5q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.031597 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.031419    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.031597 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.031447    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.031773 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.031757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.031859 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.031837    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.047772 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.047740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrl5q\" (UniqueName: \"kubernetes.io/projected/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-kube-api-access-qrl5q\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.123028 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.122964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:27.267376 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.267351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"]
Jul 16 17:43:27.269567 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:43:27.269541    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e55d6c_eeba_4ff9_8cac_f714f062c1e6.slice/crio-f9d729b0aaa72558c1e08994f77f48d7d5b85b5d30f375069ea16c3b79a2a388 WatchSource:0}: Error finding container f9d729b0aaa72558c1e08994f77f48d7d5b85b5d30f375069ea16c3b79a2a388: Status 404 returned error can't find the container with id f9d729b0aaa72558c1e08994f77f48d7d5b85b5d30f375069ea16c3b79a2a388
Jul 16 17:43:27.384541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.384513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck" event={"ID":"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6","Type":"ContainerStarted","Data":"20cef9d56c3d2c65bfd30174f123c4b1d84abd5a02123a09f72f46c9f416832d"}
Jul 16 17:43:27.384642 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:27.384567    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck" event={"ID":"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6","Type":"ContainerStarted","Data":"f9d729b0aaa72558c1e08994f77f48d7d5b85b5d30f375069ea16c3b79a2a388"}
Jul 16 17:43:28.392518 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:28.392435    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerID="20cef9d56c3d2c65bfd30174f123c4b1d84abd5a02123a09f72f46c9f416832d" exitCode=0
Jul 16 17:43:28.392871 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:28.392504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck" event={"ID":"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6","Type":"ContainerDied","Data":"20cef9d56c3d2c65bfd30174f123c4b1d84abd5a02123a09f72f46c9f416832d"}
Jul 16 17:43:30.403905 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:30.403867    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerID="0f6e61af60ae720e8728c572f68e25d007d9abdd828fa54853ca8074ce62727d" exitCode=0
Jul 16 17:43:30.404317 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:30.403951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck" event={"ID":"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6","Type":"ContainerDied","Data":"0f6e61af60ae720e8728c572f68e25d007d9abdd828fa54853ca8074ce62727d"}
Jul 16 17:43:31.411122 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:31.411036    2579 generic.go:356] "Generic (PLEG): container finished" podID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerID="67e86e984fd8d88164259a21ee97f5dc0e3a5a5024d37809e2bd620982c1ef8b" exitCode=0
Jul 16 17:43:31.411492 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:31.411123    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck" event={"ID":"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6","Type":"ContainerDied","Data":"67e86e984fd8d88164259a21ee97f5dc0e3a5a5024d37809e2bd620982c1ef8b"}
Jul 16 17:43:32.547186 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.547158    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:32.678768 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.678676    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrl5q\" (UniqueName: \"kubernetes.io/projected/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-kube-api-access-qrl5q\") pod \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") "
Jul 16 17:43:32.678908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.678779    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-bundle\") pod \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") "
Jul 16 17:43:32.678908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.678797    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-util\") pod \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\" (UID: \"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6\") "
Jul 16 17:43:32.679241 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.679207    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-bundle" (OuterVolumeSpecName: "bundle") pod "c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" (UID: "c7e55d6c-eeba-4ff9-8cac-f714f062c1e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:43:32.680924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.680901    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-kube-api-access-qrl5q" (OuterVolumeSpecName: "kube-api-access-qrl5q") pod "c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" (UID: "c7e55d6c-eeba-4ff9-8cac-f714f062c1e6"). InnerVolumeSpecName "kube-api-access-qrl5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:43:32.684873 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.684831    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-util" (OuterVolumeSpecName: "util") pod "c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" (UID: "c7e55d6c-eeba-4ff9-8cac-f714f062c1e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:43:32.780250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.780214    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:32.780250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.780242    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:32.780250 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:32.780250    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrl5q\" (UniqueName: \"kubernetes.io/projected/c7e55d6c-eeba-4ff9-8cac-f714f062c1e6-kube-api-access-qrl5q\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:33.421602 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:33.421514    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck"
Jul 16 17:43:33.421602 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:33.421536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fxnmck" event={"ID":"c7e55d6c-eeba-4ff9-8cac-f714f062c1e6","Type":"ContainerDied","Data":"f9d729b0aaa72558c1e08994f77f48d7d5b85b5d30f375069ea16c3b79a2a388"}
Jul 16 17:43:33.421602 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:33.421580    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9d729b0aaa72558c1e08994f77f48d7d5b85b5d30f375069ea16c3b79a2a388"
Jul 16 17:43:48.451212 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.451173    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5bc48db47c-x2k8n" podUID="1d662f1d-114f-4f59-bc86-8631006a338c" containerName="console" containerID="cri-o://051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5" gracePeriod=15
Jul 16 17:43:48.699787 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.699761    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:43:48.717385 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-trusted-ca-bundle\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.717385 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-service-ca\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.717538 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717415    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-console-config\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.717538 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717440    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9llwl\" (UniqueName: \"kubernetes.io/projected/1d662f1d-114f-4f59-bc86-8631006a338c-kube-api-access-9llwl\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.717538 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717466    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-serving-cert\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.717538 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-oauth-serving-cert\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.717788 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717748    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:43:48.717968 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-service-ca" (OuterVolumeSpecName: "service-ca") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:43:48.717968 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.717826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-console-config" (OuterVolumeSpecName: "console-config") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:43:48.718089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.718018    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:43:48.719542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.719520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d662f1d-114f-4f59-bc86-8631006a338c-kube-api-access-9llwl" (OuterVolumeSpecName: "kube-api-access-9llwl") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "kube-api-access-9llwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:43:48.719924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.719904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:43:48.818527 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818494    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-oauth-config\") pod \"1d662f1d-114f-4f59-bc86-8631006a338c\" (UID: \"1d662f1d-114f-4f59-bc86-8631006a338c\") "
Jul 16 17:43:48.818699 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818661    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-console-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:48.818699 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818693    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9llwl\" (UniqueName: \"kubernetes.io/projected/1d662f1d-114f-4f59-bc86-8631006a338c-kube-api-access-9llwl\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:48.818772 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818703    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:48.818772 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818713    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-oauth-serving-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:48.818772 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818720    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-trusted-ca-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:48.818772 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.818728    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d662f1d-114f-4f59-bc86-8631006a338c-service-ca\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:48.820470 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.820447    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1d662f1d-114f-4f59-bc86-8631006a338c" (UID: "1d662f1d-114f-4f59-bc86-8631006a338c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:43:48.919606 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:48.919571    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d662f1d-114f-4f59-bc86-8631006a338c-console-oauth-config\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:43:49.498829 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.498799    2579 generic.go:356] "Generic (PLEG): container finished" podID="1d662f1d-114f-4f59-bc86-8631006a338c" containerID="051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5" exitCode=0
Jul 16 17:43:49.499355 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.498845    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5bc48db47c-x2k8n"
Jul 16 17:43:49.499355 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.498891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc48db47c-x2k8n" event={"ID":"1d662f1d-114f-4f59-bc86-8631006a338c","Type":"ContainerDied","Data":"051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5"}
Jul 16 17:43:49.499355 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.498941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5bc48db47c-x2k8n" event={"ID":"1d662f1d-114f-4f59-bc86-8631006a338c","Type":"ContainerDied","Data":"a219349f220931f5f55d0f95cfdb252a7ccf8a7e669186c249fcb9fb8791fd52"}
Jul 16 17:43:49.499355 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.498964    2579 scope.go:117] "RemoveContainer" containerID="051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5"
Jul 16 17:43:49.507543 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.507527    2579 scope.go:117] "RemoveContainer" containerID="051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5"
Jul 16 17:43:49.507799 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:43:49.507782    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5\": container with ID starting with 051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5 not found: ID does not exist" containerID="051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5"
Jul 16 17:43:49.507870 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.507807    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5"} err="failed to get container status \"051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5\": rpc error: code = NotFound desc = could not find container \"051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5\": container with ID starting with 051d469c5db02dd4fc007923c667c51fd7f4f9f01772c0ddcdd764a523b720b5 not found: ID does not exist"
Jul 16 17:43:49.527133 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.527108    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5bc48db47c-x2k8n"]
Jul 16 17:43:49.530021 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:49.530003    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5bc48db47c-x2k8n"]
Jul 16 17:43:51.103072 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:43:51.103039    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d662f1d-114f-4f59-bc86-8631006a338c" path="/var/lib/kubelet/pods/1d662f1d-114f-4f59-bc86-8631006a338c/volumes"
Jul 16 17:44:02.874314 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874280    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"]
Jul 16 17:44:02.874839 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874818    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="pull"
Jul 16 17:44:02.874915 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874841    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="pull"
Jul 16 17:44:02.874915 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874887    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="util"
Jul 16 17:44:02.874915 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874897    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="util"
Jul 16 17:44:02.875062 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874919    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d662f1d-114f-4f59-bc86-8631006a338c" containerName="console"
Jul 16 17:44:02.875062 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874934    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d662f1d-114f-4f59-bc86-8631006a338c" containerName="console"
Jul 16 17:44:02.875062 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874956    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="extract"
Jul 16 17:44:02.875062 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.874964    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="extract"
Jul 16 17:44:02.875251 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.875070    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7e55d6c-eeba-4ff9-8cac-f714f062c1e6" containerName="extract"
Jul 16 17:44:02.875251 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.875086    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d662f1d-114f-4f59-bc86-8631006a338c" containerName="console"
Jul 16 17:44:02.877979 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.877961    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.881065 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.881042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-default-dockercfg-rzcg2\""
Jul 16 17:44:02.881230 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.881213    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 17:44:02.892314 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.892291    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"]
Jul 16 17:44:02.931070 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931040    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931070 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931098    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqhfk\" (UniqueName: \"kubernetes.io/projected/67837974-612e-4f3a-9fc1-e70b10efec4e-kube-api-access-xqhfk\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67837974-612e-4f3a-9fc1-e70b10efec4e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931448 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931261    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:02.931448 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:02.931326    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032417 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqhfk\" (UniqueName: \"kubernetes.io/projected/67837974-612e-4f3a-9fc1-e70b10efec4e-kube-api-access-xqhfk\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032538    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032585 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032566    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032833 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67837974-612e-4f3a-9fc1-e70b10efec4e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032833 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032833 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.032980 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.032873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-data\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.033066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.033041    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-workload-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.033121 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.033082    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-credential-socket\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.033386 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.033366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-workload-certs\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.033432 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.033366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67837974-612e-4f3a-9fc1-e70b10efec4e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.034737 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.034712    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-envoy\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.035036 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.035020    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-podinfo\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.042953 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.042927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67837974-612e-4f3a-9fc1-e70b10efec4e-istio-token\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.043182 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.043166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqhfk\" (UniqueName: \"kubernetes.io/projected/67837974-612e-4f3a-9fc1-e70b10efec4e-kube-api-access-xqhfk\") pod \"openshift-ai-inference-openshift-default-866675cbb8-qnxtj\" (UID: \"67837974-612e-4f3a-9fc1-e70b10efec4e\") " pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.188340 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.188245    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:03.329833 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.329808    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"]
Jul 16 17:44:03.331752 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:03.331728    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67837974_612e_4f3a_9fc1_e70b10efec4e.slice/crio-c4011521964860b30900a8a65646d2df7be5d972d1499e6fe88e256d1a703a80 WatchSource:0}: Error finding container c4011521964860b30900a8a65646d2df7be5d972d1499e6fe88e256d1a703a80: Status 404 returned error can't find the container with id c4011521964860b30900a8a65646d2df7be5d972d1499e6fe88e256d1a703a80
Jul 16 17:44:03.571662 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:03.571628    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" event={"ID":"67837974-612e-4f3a-9fc1-e70b10efec4e","Type":"ContainerStarted","Data":"c4011521964860b30900a8a65646d2df7be5d972d1499e6fe88e256d1a703a80"}
Jul 16 17:44:06.897134 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:06.897096    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 17:44:06.897457 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:06.897166    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 17:44:06.897457 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:06.897196    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 17:44:07.596403 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:07.596358    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" event={"ID":"67837974-612e-4f3a-9fc1-e70b10efec4e","Type":"ContainerStarted","Data":"d0119d25cd0d66f121243f0716ab9f1410ad051f682aaaf2f6430bd76d05e56e"}
Jul 16 17:44:07.623074 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:07.623018    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" podStartSLOduration=2.059927026 podStartE2EDuration="5.623001258s" totalImagesPullingTime="3.563074232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:07.621327885 +0000 UTC m=+569.004797820" watchObservedRunningTime="2026-07-16 17:44:07.623001258 +0000 UTC m=+569.006471189"
Jul 16 17:44:08.189463 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:08.189422    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:08.194249 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:08.194225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:08.601442 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:08.601410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:08.602608 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:08.602581    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:44:10.728564 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.728526    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"]
Jul 16 17:44:10.732374 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.732353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.735483 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.735463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 17:44:10.736277 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.736241    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 17:44:10.736369 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.736307    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-6nsxr\""
Jul 16 17:44:10.743838 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.743816    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"]
Jul 16 17:44:10.799758 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.799727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.799925 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.799762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.799925 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.799809    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9989n\" (UniqueName: \"kubernetes.io/projected/ee4143d8-d853-4f35-8812-547adae170f1-kube-api-access-9989n\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.831322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.831293    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"]
Jul 16 17:44:10.834815 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.834795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:10.848416 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.848395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"]
Jul 16 17:44:10.900800 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.900768    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9989n\" (UniqueName: \"kubernetes.io/projected/ee4143d8-d853-4f35-8812-547adae170f1-kube-api-access-9989n\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.900932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.900826    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:10.900932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.900858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.900932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.900877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.900932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.900898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c9m7\" (UniqueName: \"kubernetes.io/projected/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-kube-api-access-4c9m7\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:10.900932 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.900929    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:10.901206 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.901189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.901245 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.901218    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.915717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.915689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9989n\" (UniqueName: \"kubernetes.io/projected/ee4143d8-d853-4f35-8812-547adae170f1-kube-api-access-9989n\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:10.946315 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.946280    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"]
Jul 16 17:44:10.950078 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.950064    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:10.966062 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:10.966038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"]
Jul 16 17:44:11.001882 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.001824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.001882 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.001853    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.001882 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.001874    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.002088 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.001903    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c9m7\" (UniqueName: \"kubernetes.io/projected/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-kube-api-access-4c9m7\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.002088 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.002016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbq5\" (UniqueName: \"kubernetes.io/projected/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-kube-api-access-pkbq5\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.002088 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.002043    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.002279 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.002247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.002357 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.002342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.018036 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.018013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c9m7\" (UniqueName: \"kubernetes.io/projected/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-kube-api-access-4c9m7\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.032495 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.032471    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"]
Jul 16 17:44:11.036023 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.036004    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.041175 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.041157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:11.046675 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.046654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"]
Jul 16 17:44:11.105757 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.104872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.105757 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.104933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.105757 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.104970    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.105757 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.105006    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.105757 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.105051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnckg\" (UniqueName: \"kubernetes.io/projected/49bc9e01-91b8-4236-a63d-ea749ba0534a-kube-api-access-xnckg\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.105757 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.105109    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbq5\" (UniqueName: \"kubernetes.io/projected/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-kube-api-access-pkbq5\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.108433 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.106526    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.108433 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.106954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.117508 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.117486    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbq5\" (UniqueName: \"kubernetes.io/projected/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-kube-api-access-pkbq5\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.144611 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.144576    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:11.182646 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.182478    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"]
Jul 16 17:44:11.184820 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:11.184785    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4143d8_d853_4f35_8812_547adae170f1.slice/crio-3033ba9db066a7524b21a0512f4851d52f67b2af0275f12fc252dd028cc95df4 WatchSource:0}: Error finding container 3033ba9db066a7524b21a0512f4851d52f67b2af0275f12fc252dd028cc95df4: Status 404 returned error can't find the container with id 3033ba9db066a7524b21a0512f4851d52f67b2af0275f12fc252dd028cc95df4
Jul 16 17:44:11.207168 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.206866    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.207168 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.206908    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.207168 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.206953    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnckg\" (UniqueName: \"kubernetes.io/projected/49bc9e01-91b8-4236-a63d-ea749ba0534a-kube-api-access-xnckg\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.208187 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.208143    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.208319 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.208292    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.223040 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.223018    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnckg\" (UniqueName: \"kubernetes.io/projected/49bc9e01-91b8-4236-a63d-ea749ba0534a-kube-api-access-xnckg\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.258699 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.258676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:11.289451 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.289423    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"]
Jul 16 17:44:11.292823 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:11.292776    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa5e18b_c492_4659_9aee_3e03ed6b9f89.slice/crio-a4153d3aab67213105f9791249b81f7dfd71357f414fca927d32919da5f49ce8 WatchSource:0}: Error finding container a4153d3aab67213105f9791249b81f7dfd71357f414fca927d32919da5f49ce8: Status 404 returned error can't find the container with id a4153d3aab67213105f9791249b81f7dfd71357f414fca927d32919da5f49ce8
Jul 16 17:44:11.345608 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.345565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:11.406491 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.406458    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"]
Jul 16 17:44:11.407424 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:11.407394    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c0ee82_0740_4d05_ab1e_c8ab20f6900b.slice/crio-32c7817b450658bfe6685bac46d0abaed839a9f8d0a0ad22f72e02fd5b42da32 WatchSource:0}: Error finding container 32c7817b450658bfe6685bac46d0abaed839a9f8d0a0ad22f72e02fd5b42da32: Status 404 returned error can't find the container with id 32c7817b450658bfe6685bac46d0abaed839a9f8d0a0ad22f72e02fd5b42da32
Jul 16 17:44:11.510439 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.510407    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"]
Jul 16 17:44:11.514729 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:11.514706    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bc9e01_91b8_4236_a63d_ea749ba0534a.slice/crio-f707ab701f0a62332444f8f8284105d340e170ce691b9606bf01cde2db5b101f WatchSource:0}: Error finding container f707ab701f0a62332444f8f8284105d340e170ce691b9606bf01cde2db5b101f: Status 404 returned error can't find the container with id f707ab701f0a62332444f8f8284105d340e170ce691b9606bf01cde2db5b101f
Jul 16 17:44:11.616317 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.616283    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee4143d8-d853-4f35-8812-547adae170f1" containerID="e424cd348565e225d9d8a2e8972305de79522f20483150008a07b58cfaa4c69e" exitCode=0
Jul 16 17:44:11.616494 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.616379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw" event={"ID":"ee4143d8-d853-4f35-8812-547adae170f1","Type":"ContainerDied","Data":"e424cd348565e225d9d8a2e8972305de79522f20483150008a07b58cfaa4c69e"}
Jul 16 17:44:11.616494 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.616424    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw" event={"ID":"ee4143d8-d853-4f35-8812-547adae170f1","Type":"ContainerStarted","Data":"3033ba9db066a7524b21a0512f4851d52f67b2af0275f12fc252dd028cc95df4"}
Jul 16 17:44:11.617997 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.617976    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerID="937e04c3da3e4e9e2521cd98a1b41fd08e1dad72669ed202e00767e8b34ca148" exitCode=0
Jul 16 17:44:11.618099 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.618071    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt" event={"ID":"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b","Type":"ContainerDied","Data":"937e04c3da3e4e9e2521cd98a1b41fd08e1dad72669ed202e00767e8b34ca148"}
Jul 16 17:44:11.618170 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.618125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt" event={"ID":"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b","Type":"ContainerStarted","Data":"32c7817b450658bfe6685bac46d0abaed839a9f8d0a0ad22f72e02fd5b42da32"}
Jul 16 17:44:11.619687 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.619652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv" event={"ID":"49bc9e01-91b8-4236-a63d-ea749ba0534a","Type":"ContainerStarted","Data":"4697ea4f15859d2091dc1831efb62e902e0dc227da9c9cfc634640cad7c0ea65"}
Jul 16 17:44:11.619788 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.619690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv" event={"ID":"49bc9e01-91b8-4236-a63d-ea749ba0534a","Type":"ContainerStarted","Data":"f707ab701f0a62332444f8f8284105d340e170ce691b9606bf01cde2db5b101f"}
Jul 16 17:44:11.621506 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.621482    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerID="8e8589ebca12608956062a54ac607d66da15f894bd8b4aa1492999fe7b2b1081" exitCode=0
Jul 16 17:44:11.621609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.621549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd" event={"ID":"8fa5e18b-c492-4659-9aee-3e03ed6b9f89","Type":"ContainerDied","Data":"8e8589ebca12608956062a54ac607d66da15f894bd8b4aa1492999fe7b2b1081"}
Jul 16 17:44:11.621609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:11.621580    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd" event={"ID":"8fa5e18b-c492-4659-9aee-3e03ed6b9f89","Type":"ContainerStarted","Data":"a4153d3aab67213105f9791249b81f7dfd71357f414fca927d32919da5f49ce8"}
Jul 16 17:44:12.630997 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:12.630961    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw" event={"ID":"ee4143d8-d853-4f35-8812-547adae170f1","Type":"ContainerStarted","Data":"90525b9ffe1538bd8889ca0c0dedbaf5c9a171bfc22e2f328d9bb2ae7fb5c7f7"}
Jul 16 17:44:12.633107 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:12.633075    2579 generic.go:356] "Generic (PLEG): container finished" podID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerID="4697ea4f15859d2091dc1831efb62e902e0dc227da9c9cfc634640cad7c0ea65" exitCode=0
Jul 16 17:44:12.633107 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:12.633100    2579 generic.go:356] "Generic (PLEG): container finished" podID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerID="beb753898a3e7c7c3e76c633429a9071bd2d2d61ad512b2f97ea72ea27e8a7c6" exitCode=0
Jul 16 17:44:12.633292 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:12.633155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv" event={"ID":"49bc9e01-91b8-4236-a63d-ea749ba0534a","Type":"ContainerDied","Data":"4697ea4f15859d2091dc1831efb62e902e0dc227da9c9cfc634640cad7c0ea65"}
Jul 16 17:44:12.633292 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:12.633204    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv" event={"ID":"49bc9e01-91b8-4236-a63d-ea749ba0534a","Type":"ContainerDied","Data":"beb753898a3e7c7c3e76c633429a9071bd2d2d61ad512b2f97ea72ea27e8a7c6"}
Jul 16 17:44:13.641182 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:13.641148    2579 generic.go:356] "Generic (PLEG): container finished" podID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerID="0864b46450e008c1e5589b777dba1d4daabf68e286aec2cf8f96d305d7c31a0c" exitCode=0
Jul 16 17:44:13.641625 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:13.641229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv" event={"ID":"49bc9e01-91b8-4236-a63d-ea749ba0534a","Type":"ContainerDied","Data":"0864b46450e008c1e5589b777dba1d4daabf68e286aec2cf8f96d305d7c31a0c"}
Jul 16 17:44:13.643035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:13.643014    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee4143d8-d853-4f35-8812-547adae170f1" containerID="90525b9ffe1538bd8889ca0c0dedbaf5c9a171bfc22e2f328d9bb2ae7fb5c7f7" exitCode=0
Jul 16 17:44:13.643162 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:13.643082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw" event={"ID":"ee4143d8-d853-4f35-8812-547adae170f1","Type":"ContainerDied","Data":"90525b9ffe1538bd8889ca0c0dedbaf5c9a171bfc22e2f328d9bb2ae7fb5c7f7"}
Jul 16 17:44:13.644868 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:13.644848    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerID="8db22c94014e9029dcc015dd7cc2185eef54614c564ebe05a3edbcafb6cadaa8" exitCode=0
Jul 16 17:44:13.644953 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:13.644886    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt" event={"ID":"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b","Type":"ContainerDied","Data":"8db22c94014e9029dcc015dd7cc2185eef54614c564ebe05a3edbcafb6cadaa8"}
Jul 16 17:44:14.651606 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.651567    2579 generic.go:356] "Generic (PLEG): container finished" podID="ee4143d8-d853-4f35-8812-547adae170f1" containerID="73e4f64ce1f02cf7af1661307146fd1915aa319e220ecbd0af2675a14ca9ea34" exitCode=0
Jul 16 17:44:14.652044 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.651654    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw" event={"ID":"ee4143d8-d853-4f35-8812-547adae170f1","Type":"ContainerDied","Data":"73e4f64ce1f02cf7af1661307146fd1915aa319e220ecbd0af2675a14ca9ea34"}
Jul 16 17:44:14.653591 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.653571    2579 generic.go:356] "Generic (PLEG): container finished" podID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerID="177ec6aeba250cdc30fae336db28114e999ce4ce2c4a2f0c6166fdb2b84486b0" exitCode=0
Jul 16 17:44:14.653706 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.653630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt" event={"ID":"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b","Type":"ContainerDied","Data":"177ec6aeba250cdc30fae336db28114e999ce4ce2c4a2f0c6166fdb2b84486b0"}
Jul 16 17:44:14.784083 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.784058    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:14.837679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.837644    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-bundle\") pod \"49bc9e01-91b8-4236-a63d-ea749ba0534a\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") "
Jul 16 17:44:14.837679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.837672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-util\") pod \"49bc9e01-91b8-4236-a63d-ea749ba0534a\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") "
Jul 16 17:44:14.837910 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.837726    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnckg\" (UniqueName: \"kubernetes.io/projected/49bc9e01-91b8-4236-a63d-ea749ba0534a-kube-api-access-xnckg\") pod \"49bc9e01-91b8-4236-a63d-ea749ba0534a\" (UID: \"49bc9e01-91b8-4236-a63d-ea749ba0534a\") "
Jul 16 17:44:14.838147 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.838115    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-bundle" (OuterVolumeSpecName: "bundle") pod "49bc9e01-91b8-4236-a63d-ea749ba0534a" (UID: "49bc9e01-91b8-4236-a63d-ea749ba0534a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:14.839908 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.839883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49bc9e01-91b8-4236-a63d-ea749ba0534a-kube-api-access-xnckg" (OuterVolumeSpecName: "kube-api-access-xnckg") pod "49bc9e01-91b8-4236-a63d-ea749ba0534a" (UID: "49bc9e01-91b8-4236-a63d-ea749ba0534a"). InnerVolumeSpecName "kube-api-access-xnckg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:44:14.845184 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.845140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-util" (OuterVolumeSpecName: "util") pod "49bc9e01-91b8-4236-a63d-ea749ba0534a" (UID: "49bc9e01-91b8-4236-a63d-ea749ba0534a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:14.938768 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.938689    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnckg\" (UniqueName: \"kubernetes.io/projected/49bc9e01-91b8-4236-a63d-ea749ba0534a-kube-api-access-xnckg\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:14.938768 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.938719    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:14.938768 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:14.938728    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49bc9e01-91b8-4236-a63d-ea749ba0534a-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:15.660508 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.660472    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd" event={"ID":"8fa5e18b-c492-4659-9aee-3e03ed6b9f89","Type":"ContainerStarted","Data":"24b6e18ef7771275b596ef70049f20b3baec49881ba6bb4c6216357dd0695551"}
Jul 16 17:44:15.662560 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.662542    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv"
Jul 16 17:44:15.662670 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.662586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36mmhnv" event={"ID":"49bc9e01-91b8-4236-a63d-ea749ba0534a","Type":"ContainerDied","Data":"f707ab701f0a62332444f8f8284105d340e170ce691b9606bf01cde2db5b101f"}
Jul 16 17:44:15.662670 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.662637    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f707ab701f0a62332444f8f8284105d340e170ce691b9606bf01cde2db5b101f"
Jul 16 17:44:15.812859 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.812800    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:15.827444 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.827418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:15.946622 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.946590    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-bundle\") pod \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") "
Jul 16 17:44:15.946770 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.946635    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkbq5\" (UniqueName: \"kubernetes.io/projected/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-kube-api-access-pkbq5\") pod \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") "
Jul 16 17:44:15.946770 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.946672    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-util\") pod \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\" (UID: \"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b\") "
Jul 16 17:44:15.946770 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.946697    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9989n\" (UniqueName: \"kubernetes.io/projected/ee4143d8-d853-4f35-8812-547adae170f1-kube-api-access-9989n\") pod \"ee4143d8-d853-4f35-8812-547adae170f1\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") "
Jul 16 17:44:15.946770 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.946719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-util\") pod \"ee4143d8-d853-4f35-8812-547adae170f1\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") "
Jul 16 17:44:15.946770 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.946754    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-bundle\") pod \"ee4143d8-d853-4f35-8812-547adae170f1\" (UID: \"ee4143d8-d853-4f35-8812-547adae170f1\") "
Jul 16 17:44:15.947443 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.947390    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-bundle" (OuterVolumeSpecName: "bundle") pod "ee4143d8-d853-4f35-8812-547adae170f1" (UID: "ee4143d8-d853-4f35-8812-547adae170f1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:15.947586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.947471    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-bundle" (OuterVolumeSpecName: "bundle") pod "e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" (UID: "e8c0ee82-0740-4d05-ab1e-c8ab20f6900b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:15.948888 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.948864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-kube-api-access-pkbq5" (OuterVolumeSpecName: "kube-api-access-pkbq5") pod "e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" (UID: "e8c0ee82-0740-4d05-ab1e-c8ab20f6900b"). InnerVolumeSpecName "kube-api-access-pkbq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:44:15.949001 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.948909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4143d8-d853-4f35-8812-547adae170f1-kube-api-access-9989n" (OuterVolumeSpecName: "kube-api-access-9989n") pod "ee4143d8-d853-4f35-8812-547adae170f1" (UID: "ee4143d8-d853-4f35-8812-547adae170f1"). InnerVolumeSpecName "kube-api-access-9989n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:44:15.952872 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.952850    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-util" (OuterVolumeSpecName: "util") pod "ee4143d8-d853-4f35-8812-547adae170f1" (UID: "ee4143d8-d853-4f35-8812-547adae170f1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:15.952945 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:15.952868    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-util" (OuterVolumeSpecName: "util") pod "e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" (UID: "e8c0ee82-0740-4d05-ab1e-c8ab20f6900b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:16.047434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.047399    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:16.047434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.047429    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9989n\" (UniqueName: \"kubernetes.io/projected/ee4143d8-d853-4f35-8812-547adae170f1-kube-api-access-9989n\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:16.047434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.047441    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:16.047635 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.047449    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4143d8-d853-4f35-8812-547adae170f1-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:16.047635 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.047458    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:16.047635 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.047466    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkbq5\" (UniqueName: \"kubernetes.io/projected/e8c0ee82-0740-4d05-ab1e-c8ab20f6900b-kube-api-access-pkbq5\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:16.265715 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.265640    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"]
Jul 16 17:44:16.266004 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.265992    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="pull"
Jul 16 17:44:16.266047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266006    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="pull"
Jul 16 17:44:16.266047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266018    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="pull"
Jul 16 17:44:16.266047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266023    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="pull"
Jul 16 17:44:16.266047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266031    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="extract"
Jul 16 17:44:16.266047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266038    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="extract"
Jul 16 17:44:16.266047 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266046    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="extract"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="extract"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266065    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="util"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266070    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="util"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266080    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="util"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266085    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="util"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266093    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="util"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266098    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="util"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266104    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="pull"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266108    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="pull"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266113    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="extract"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266117    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="extract"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266171    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c0ee82-0740-4d05-ab1e-c8ab20f6900b" containerName="extract"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266178    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="49bc9e01-91b8-4236-a63d-ea749ba0534a" containerName="extract"
Jul 16 17:44:16.266218 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.266184    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee4143d8-d853-4f35-8812-547adae170f1" containerName="extract"
Jul 16 17:44:16.270371 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.270352    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.274396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.274370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 17:44:16.274532 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.274418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-77dwh\""
Jul 16 17:44:16.275398 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.275382    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 17:44:16.275886 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.275870    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 17:44:16.276074 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.276060    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:44:16.276455 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.276444    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 17:44:16.288071 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.288050    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"]
Jul 16 17:44:16.349402 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.349374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57ad18f4-aef4-4593-9809-a6d2134f458f-metrics-cert\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.349554 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.349440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/57ad18f4-aef4-4593-9809-a6d2134f458f-manager-config\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.349554 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.349472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57ad18f4-aef4-4593-9809-a6d2134f458f-cert\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.349554 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.349499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5ngr\" (UniqueName: \"kubernetes.io/projected/57ad18f4-aef4-4593-9809-a6d2134f458f-kube-api-access-r5ngr\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.450863 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.450830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57ad18f4-aef4-4593-9809-a6d2134f458f-metrics-cert\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.450993 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.450890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/57ad18f4-aef4-4593-9809-a6d2134f458f-manager-config\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.450993 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.450916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57ad18f4-aef4-4593-9809-a6d2134f458f-cert\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.450993 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.450937    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5ngr\" (UniqueName: \"kubernetes.io/projected/57ad18f4-aef4-4593-9809-a6d2134f458f-kube-api-access-r5ngr\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.451603 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.451579    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/57ad18f4-aef4-4593-9809-a6d2134f458f-manager-config\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.453313 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.453291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57ad18f4-aef4-4593-9809-a6d2134f458f-cert\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.453389 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.453342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/57ad18f4-aef4-4593-9809-a6d2134f458f-metrics-cert\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.556523 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.556500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5ngr\" (UniqueName: \"kubernetes.io/projected/57ad18f4-aef4-4593-9809-a6d2134f458f-kube-api-access-r5ngr\") pod \"lws-controller-manager-5c87b7c6dd-v85f7\" (UID: \"57ad18f4-aef4-4593-9809-a6d2134f458f\") " pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.579308 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.579286    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:16.672627 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.672576    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerID="24b6e18ef7771275b596ef70049f20b3baec49881ba6bb4c6216357dd0695551" exitCode=0
Jul 16 17:44:16.673176 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.672733    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd" event={"ID":"8fa5e18b-c492-4659-9aee-3e03ed6b9f89","Type":"ContainerDied","Data":"24b6e18ef7771275b596ef70049f20b3baec49881ba6bb4c6216357dd0695551"}
Jul 16 17:44:16.675048 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.675025    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw"
Jul 16 17:44:16.675154 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.675028    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efgrhbw" event={"ID":"ee4143d8-d853-4f35-8812-547adae170f1","Type":"ContainerDied","Data":"3033ba9db066a7524b21a0512f4851d52f67b2af0275f12fc252dd028cc95df4"}
Jul 16 17:44:16.675154 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.675147    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3033ba9db066a7524b21a0512f4851d52f67b2af0275f12fc252dd028cc95df4"
Jul 16 17:44:16.678633 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.677048    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt" event={"ID":"e8c0ee82-0740-4d05-ab1e-c8ab20f6900b","Type":"ContainerDied","Data":"32c7817b450658bfe6685bac46d0abaed839a9f8d0a0ad22f72e02fd5b42da32"}
Jul 16 17:44:16.678633 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.677088    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c7817b450658bfe6685bac46d0abaed839a9f8d0a0ad22f72e02fd5b42da32"
Jul 16 17:44:16.678633 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.677095    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76rxdrt"
Jul 16 17:44:16.812206 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:16.812184    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"]
Jul 16 17:44:16.814232 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:16.814146    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ad18f4_aef4_4593_9809_a6d2134f458f.slice/crio-66e5cd782157b3e19c8e7c6c82147c90591998c7d81d72dcc2e633e1475bc056 WatchSource:0}: Error finding container 66e5cd782157b3e19c8e7c6c82147c90591998c7d81d72dcc2e633e1475bc056: Status 404 returned error can't find the container with id 66e5cd782157b3e19c8e7c6c82147c90591998c7d81d72dcc2e633e1475bc056
Jul 16 17:44:17.689013 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:17.688975    2579 generic.go:356] "Generic (PLEG): container finished" podID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerID="0a90b0905b5d69642471b6991019db975e5f7d60dc7a512950e6b8f5cfe73a4a" exitCode=0
Jul 16 17:44:17.689489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:17.689059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd" event={"ID":"8fa5e18b-c492-4659-9aee-3e03ed6b9f89","Type":"ContainerDied","Data":"0a90b0905b5d69642471b6991019db975e5f7d60dc7a512950e6b8f5cfe73a4a"}
Jul 16 17:44:17.690626 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:17.690600    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7" event={"ID":"57ad18f4-aef4-4593-9809-a6d2134f458f","Type":"ContainerStarted","Data":"66e5cd782157b3e19c8e7c6c82147c90591998c7d81d72dcc2e633e1475bc056"}
Jul 16 17:44:18.698578 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.698483    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7" event={"ID":"57ad18f4-aef4-4593-9809-a6d2134f458f","Type":"ContainerStarted","Data":"d7ea529e7c0ca7bbc7c735fe4bd11136ddfb11c0b9b0f1464f141d96549d9267"}
Jul 16 17:44:18.698992 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.698608    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:18.723933 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.723882    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7" podStartSLOduration=1.155323613 podStartE2EDuration="2.723863841s" totalImagesPullingTime="1.568540228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:18.722767032 +0000 UTC m=+580.106236968" watchObservedRunningTime="2026-07-16 17:44:18.723863841 +0000 UTC m=+580.107333774"
Jul 16 17:44:18.833475 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.833445    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:18.975620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.975536    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4c9m7\" (UniqueName: \"kubernetes.io/projected/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-kube-api-access-4c9m7\") pod \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") "
Jul 16 17:44:18.975620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.975589    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-bundle\") pod \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") "
Jul 16 17:44:18.975620 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.975624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-util\") pod \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\" (UID: \"8fa5e18b-c492-4659-9aee-3e03ed6b9f89\") "
Jul 16 17:44:18.976167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.976130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-bundle" (OuterVolumeSpecName: "bundle") pod "8fa5e18b-c492-4659-9aee-3e03ed6b9f89" (UID: "8fa5e18b-c492-4659-9aee-3e03ed6b9f89"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:18.977712 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.977691    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-kube-api-access-4c9m7" (OuterVolumeSpecName: "kube-api-access-4c9m7") pod "8fa5e18b-c492-4659-9aee-3e03ed6b9f89" (UID: "8fa5e18b-c492-4659-9aee-3e03ed6b9f89"). InnerVolumeSpecName "kube-api-access-4c9m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:44:18.980377 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:18.980355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-util" (OuterVolumeSpecName: "util") pod "8fa5e18b-c492-4659-9aee-3e03ed6b9f89" (UID: "8fa5e18b-c492-4659-9aee-3e03ed6b9f89"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:44:19.076323 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:19.076258    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c9m7\" (UniqueName: \"kubernetes.io/projected/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-kube-api-access-4c9m7\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:19.076323 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:19.076320    2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-bundle\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:19.076323 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:19.076331    2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8fa5e18b-c492-4659-9aee-3e03ed6b9f89-util\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:44:19.705735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:19.705648    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd"
Jul 16 17:44:19.705735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:19.705685    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabzpmrd" event={"ID":"8fa5e18b-c492-4659-9aee-3e03ed6b9f89","Type":"ContainerDied","Data":"a4153d3aab67213105f9791249b81f7dfd71357f414fca927d32919da5f49ce8"}
Jul 16 17:44:19.705735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:19.705731    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4153d3aab67213105f9791249b81f7dfd71357f414fca927d32919da5f49ce8"
Jul 16 17:44:29.708233 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:29.708201    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c87b7c6dd-v85f7"
Jul 16 17:44:38.999399 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:38.999364    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:44:39.000536 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:39.000516    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:44:40.126744 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.126710    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"]
Jul 16 17:44:40.127116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127058    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="extract"
Jul 16 17:44:40.127116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127068    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="extract"
Jul 16 17:44:40.127116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127078    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="pull"
Jul 16 17:44:40.127116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="pull"
Jul 16 17:44:40.127116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127099    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="util"
Jul 16 17:44:40.127116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127105    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="util"
Jul 16 17:44:40.127359 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.127162    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fa5e18b-c492-4659-9aee-3e03ed6b9f89" containerName="extract"
Jul 16 17:44:40.136966 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.136947    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.139858 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.139836    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 16 17:44:40.140202 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.140182    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 17:44:40.140329 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.140205    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 16 17:44:40.141226 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.141210    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 17:44:40.141348 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.141234    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-68tkn\""
Jul 16 17:44:40.153067 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.153047    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"]
Jul 16 17:44:40.260886 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.260852    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.261067 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.260901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.261067 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.260919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t8fk\" (UniqueName: \"kubernetes.io/projected/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-kube-api-access-2t8fk\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.362240 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.362209    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.362240 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.362238    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t8fk\" (UniqueName: \"kubernetes.io/projected/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-kube-api-access-2t8fk\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.362456 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.362319    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.362456 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:44:40.362362    2579 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 16 17:44:40.362456 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:44:40.362432    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-plugin-serving-cert podName:4b53f64d-35e4-4de9-98c6-9ad9f369b8bc nodeName:}" failed. No retries permitted until 2026-07-16 17:44:40.862414588 +0000 UTC m=+602.245884514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-plugin-serving-cert") pod "kuadrant-console-plugin-85fd4b8b9d-htdmf" (UID: "4b53f64d-35e4-4de9-98c6-9ad9f369b8bc") : secret "plugin-serving-cert" not found
Jul 16 17:44:40.362890 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.362873    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-nginx-conf\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.375801 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.375768    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t8fk\" (UniqueName: \"kubernetes.io/projected/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-kube-api-access-2t8fk\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.866387 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.866353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:40.868807 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:40.868785    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4b53f64d-35e4-4de9-98c6-9ad9f369b8bc-plugin-serving-cert\") pod \"kuadrant-console-plugin-85fd4b8b9d-htdmf\" (UID: \"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc\") " pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:41.059602 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:41.059573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"
Jul 16 17:44:41.195163 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:41.195135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf"]
Jul 16 17:44:41.197289 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:44:41.197241    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b53f64d_35e4_4de9_98c6_9ad9f369b8bc.slice/crio-321d5ae2e763a815a8300f89179d861b88be3607ef26d63b9c0031e60643b5be WatchSource:0}: Error finding container 321d5ae2e763a815a8300f89179d861b88be3607ef26d63b9c0031e60643b5be: Status 404 returned error can't find the container with id 321d5ae2e763a815a8300f89179d861b88be3607ef26d63b9c0031e60643b5be
Jul 16 17:44:41.818149 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:41.818112    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf" event={"ID":"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc","Type":"ContainerStarted","Data":"321d5ae2e763a815a8300f89179d861b88be3607ef26d63b9c0031e60643b5be"}
Jul 16 17:44:46.848756 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:46.848664    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf" event={"ID":"4b53f64d-35e4-4de9-98c6-9ad9f369b8bc","Type":"ContainerStarted","Data":"c764d780b7d0371aa467332e082cf3469fd8d98f7c3833d13a04780e8a4a0c67"}
Jul 16 17:44:46.869281 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:44:46.869212    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-85fd4b8b9d-htdmf" podStartSLOduration=1.498946717 podStartE2EDuration="6.86919214s" totalImagesPullingTime="5.370245423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:44:46.868933096 +0000 UTC m=+608.252403031" watchObservedRunningTime="2026-07-16 17:44:46.86919214 +0000 UTC m=+608.252662071"
Jul 16 17:45:22.663162 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.663127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b6c89bb85-l94lp"]
Jul 16 17:45:22.666588 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.666569    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:22.669847 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.669822    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-4pgm6\""
Jul 16 17:45:22.675844 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.675821    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-l94lp"]
Jul 16 17:45:22.810783 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.810750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt985\" (UniqueName: \"kubernetes.io/projected/5fa0b8ce-197f-4823-9f1b-76d1578ad078-kube-api-access-qt985\") pod \"authorino-b6c89bb85-l94lp\" (UID: \"5fa0b8ce-197f-4823-9f1b-76d1578ad078\") " pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:22.911548 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.911518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt985\" (UniqueName: \"kubernetes.io/projected/5fa0b8ce-197f-4823-9f1b-76d1578ad078-kube-api-access-qt985\") pod \"authorino-b6c89bb85-l94lp\" (UID: \"5fa0b8ce-197f-4823-9f1b-76d1578ad078\") " pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:22.921131 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.921072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt985\" (UniqueName: \"kubernetes.io/projected/5fa0b8ce-197f-4823-9f1b-76d1578ad078-kube-api-access-qt985\") pod \"authorino-b6c89bb85-l94lp\" (UID: \"5fa0b8ce-197f-4823-9f1b-76d1578ad078\") " pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:22.976169 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:22.976140    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:23.125570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:23.125539    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-l94lp"]
Jul 16 17:45:23.128826 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:45:23.128796    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fa0b8ce_197f_4823_9f1b_76d1578ad078.slice/crio-fb3456c2b16ee6134dafee3670e9bd0cfcc73af3f4ce6dfa6ad0fdc22cf595a5 WatchSource:0}: Error finding container fb3456c2b16ee6134dafee3670e9bd0cfcc73af3f4ce6dfa6ad0fdc22cf595a5: Status 404 returned error can't find the container with id fb3456c2b16ee6134dafee3670e9bd0cfcc73af3f4ce6dfa6ad0fdc22cf595a5
Jul 16 17:45:24.046007 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:24.045964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-l94lp" event={"ID":"5fa0b8ce-197f-4823-9f1b-76d1578ad078","Type":"ContainerStarted","Data":"fb3456c2b16ee6134dafee3670e9bd0cfcc73af3f4ce6dfa6ad0fdc22cf595a5"}
Jul 16 17:45:25.052626 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:25.052588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-l94lp" event={"ID":"5fa0b8ce-197f-4823-9f1b-76d1578ad078","Type":"ContainerStarted","Data":"1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900"}
Jul 16 17:45:25.070225 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:25.070178    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b6c89bb85-l94lp" podStartSLOduration=1.367287023 podStartE2EDuration="3.070162202s" totalImagesPullingTime="1.702875179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:25.06981712 +0000 UTC m=+646.453287057" watchObservedRunningTime="2026-07-16 17:45:25.070162202 +0000 UTC m=+646.453632178"
Jul 16 17:45:27.623608 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:27.623570    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-l94lp"]
Jul 16 17:45:27.624061 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:27.623789    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b6c89bb85-l94lp" podUID="5fa0b8ce-197f-4823-9f1b-76d1578ad078" containerName="authorino" containerID="cri-o://1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900" gracePeriod=30
Jul 16 17:45:27.866928 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:27.866898    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:27.957489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:27.957409    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qt985\" (UniqueName: \"kubernetes.io/projected/5fa0b8ce-197f-4823-9f1b-76d1578ad078-kube-api-access-qt985\") pod \"5fa0b8ce-197f-4823-9f1b-76d1578ad078\" (UID: \"5fa0b8ce-197f-4823-9f1b-76d1578ad078\") "
Jul 16 17:45:27.959508 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:27.959477    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa0b8ce-197f-4823-9f1b-76d1578ad078-kube-api-access-qt985" (OuterVolumeSpecName: "kube-api-access-qt985") pod "5fa0b8ce-197f-4823-9f1b-76d1578ad078" (UID: "5fa0b8ce-197f-4823-9f1b-76d1578ad078"). InnerVolumeSpecName "kube-api-access-qt985". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:45:28.058595 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.058564    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt985\" (UniqueName: \"kubernetes.io/projected/5fa0b8ce-197f-4823-9f1b-76d1578ad078-kube-api-access-qt985\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:45:28.069140 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.069110    2579 generic.go:356] "Generic (PLEG): container finished" podID="5fa0b8ce-197f-4823-9f1b-76d1578ad078" containerID="1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900" exitCode=0
Jul 16 17:45:28.069278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.069149    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-l94lp" event={"ID":"5fa0b8ce-197f-4823-9f1b-76d1578ad078","Type":"ContainerDied","Data":"1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900"}
Jul 16 17:45:28.069278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.069151    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b6c89bb85-l94lp"
Jul 16 17:45:28.069278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.069174    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b6c89bb85-l94lp" event={"ID":"5fa0b8ce-197f-4823-9f1b-76d1578ad078","Type":"ContainerDied","Data":"fb3456c2b16ee6134dafee3670e9bd0cfcc73af3f4ce6dfa6ad0fdc22cf595a5"}
Jul 16 17:45:28.069278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.069190    2579 scope.go:117] "RemoveContainer" containerID="1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900"
Jul 16 17:45:28.078718 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.078701    2579 scope.go:117] "RemoveContainer" containerID="1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900"
Jul 16 17:45:28.078947 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:45:28.078932    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900\": container with ID starting with 1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900 not found: ID does not exist" containerID="1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900"
Jul 16 17:45:28.078988 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.078954    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900"} err="failed to get container status \"1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900\": rpc error: code = NotFound desc = could not find container \"1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900\": container with ID starting with 1e5e5fe83ad85c8044d25a24c02629700f8128b8e9bfa0a53f4e87c10aa4d900 not found: ID does not exist"
Jul 16 17:45:28.094115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.094093    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-l94lp"]
Jul 16 17:45:28.099526 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:28.099507    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b6c89bb85-l94lp"]
Jul 16 17:45:29.103017 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:45:29.102985    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fa0b8ce-197f-4823-9f1b-76d1578ad078" path="/var/lib/kubelet/pods/5fa0b8ce-197f-4823-9f1b-76d1578ad078/volumes"
Jul 16 17:46:45.790242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.790165    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-4sjxq"]
Jul 16 17:46:45.790823 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.790515    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fa0b8ce-197f-4823-9f1b-76d1578ad078" containerName="authorino"
Jul 16 17:46:45.790823 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.790526    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fa0b8ce-197f-4823-9f1b-76d1578ad078" containerName="authorino"
Jul 16 17:46:45.790823 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.790597    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fa0b8ce-197f-4823-9f1b-76d1578ad078" containerName="authorino"
Jul 16 17:46:45.793515 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.793500    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4sjxq"
Jul 16 17:46:45.796572 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.796547    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 16 17:46:45.796870 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.796845    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-52rtb\""
Jul 16 17:46:45.806883 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.806859    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4sjxq"]
Jul 16 17:46:45.820627 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.820607    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttgt\" (UniqueName: \"kubernetes.io/projected/34e08602-8875-4c94-9f8b-07a239066a8e-kube-api-access-jttgt\") pod \"s3-init-4sjxq\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") " pod="kserve/s3-init-4sjxq"
Jul 16 17:46:45.820735 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.820651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e08602-8875-4c94-9f8b-07a239066a8e-model-cache\") pod \"s3-init-4sjxq\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") " pod="kserve/s3-init-4sjxq"
Jul 16 17:46:45.921288 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.921240    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttgt\" (UniqueName: \"kubernetes.io/projected/34e08602-8875-4c94-9f8b-07a239066a8e-kube-api-access-jttgt\") pod \"s3-init-4sjxq\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") " pod="kserve/s3-init-4sjxq"
Jul 16 17:46:45.921437 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.921308    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e08602-8875-4c94-9f8b-07a239066a8e-model-cache\") pod \"s3-init-4sjxq\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") " pod="kserve/s3-init-4sjxq"
Jul 16 17:46:45.921630 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.921614    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e08602-8875-4c94-9f8b-07a239066a8e-model-cache\") pod \"s3-init-4sjxq\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") " pod="kserve/s3-init-4sjxq"
Jul 16 17:46:45.931138 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:45.931106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttgt\" (UniqueName: \"kubernetes.io/projected/34e08602-8875-4c94-9f8b-07a239066a8e-kube-api-access-jttgt\") pod \"s3-init-4sjxq\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") " pod="kserve/s3-init-4sjxq"
Jul 16 17:46:46.109340 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:46.109229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4sjxq"
Jul 16 17:46:46.238699 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:46.238675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-4sjxq"]
Jul 16 17:46:46.240649 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:46:46.240619    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34e08602_8875_4c94_9f8b_07a239066a8e.slice/crio-98f0374ba59973e5c16b09f868d5bae2ae4990f8e627c995478af37e2f9e1276 WatchSource:0}: Error finding container 98f0374ba59973e5c16b09f868d5bae2ae4990f8e627c995478af37e2f9e1276: Status 404 returned error can't find the container with id 98f0374ba59973e5c16b09f868d5bae2ae4990f8e627c995478af37e2f9e1276
Jul 16 17:46:46.242562 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:46.242545    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:46:46.458730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:46.458650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4sjxq" event={"ID":"34e08602-8875-4c94-9f8b-07a239066a8e","Type":"ContainerStarted","Data":"98f0374ba59973e5c16b09f868d5bae2ae4990f8e627c995478af37e2f9e1276"}
Jul 16 17:46:50.485487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:50.485436    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4sjxq" event={"ID":"34e08602-8875-4c94-9f8b-07a239066a8e","Type":"ContainerStarted","Data":"8c2c322be745765cea36c5e8a0fea4d1911d09a99f895192436b060a7f9dedd6"}
Jul 16 17:46:51.492491 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:51.492456    2579 generic.go:356] "Generic (PLEG): container finished" podID="34e08602-8875-4c94-9f8b-07a239066a8e" containerID="8c2c322be745765cea36c5e8a0fea4d1911d09a99f895192436b060a7f9dedd6" exitCode=0
Jul 16 17:46:51.492867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:51.492543    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4sjxq" event={"ID":"34e08602-8875-4c94-9f8b-07a239066a8e","Type":"ContainerDied","Data":"8c2c322be745765cea36c5e8a0fea4d1911d09a99f895192436b060a7f9dedd6"}
Jul 16 17:46:56.521983 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:56.521948    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4sjxq" event={"ID":"34e08602-8875-4c94-9f8b-07a239066a8e","Type":"ContainerStarted","Data":"3661f80a47bcfaf538576dc22872afc689c09df6de10077b7d3575ec449314f4"}
Jul 16 17:46:56.543134 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:46:56.543085    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-4sjxq" podStartSLOduration=2.76260598 podStartE2EDuration="11.543070402s" totalImagesPullingTime="7.662244379s" totalInitContainerRuntime="1.118220043s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:56.541404667 +0000 UTC m=+737.924874600" watchObservedRunningTime="2026-07-16 17:46:56.543070402 +0000 UTC m=+737.926540335"
Jul 16 17:48:00.844115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:00.844035    2579 generic.go:356] "Generic (PLEG): container finished" podID="34e08602-8875-4c94-9f8b-07a239066a8e" containerID="3661f80a47bcfaf538576dc22872afc689c09df6de10077b7d3575ec449314f4" exitCode=0
Jul 16 17:48:00.844526 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:00.844146    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4sjxq" event={"ID":"34e08602-8875-4c94-9f8b-07a239066a8e","Type":"ContainerDied","Data":"3661f80a47bcfaf538576dc22872afc689c09df6de10077b7d3575ec449314f4"}
Jul 16 17:48:01.981228 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:01.981200    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4sjxq"
Jul 16 17:48:02.094644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.094604    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jttgt\" (UniqueName: \"kubernetes.io/projected/34e08602-8875-4c94-9f8b-07a239066a8e-kube-api-access-jttgt\") pod \"34e08602-8875-4c94-9f8b-07a239066a8e\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") "
Jul 16 17:48:02.094828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.094719    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e08602-8875-4c94-9f8b-07a239066a8e-model-cache\") pod \"34e08602-8875-4c94-9f8b-07a239066a8e\" (UID: \"34e08602-8875-4c94-9f8b-07a239066a8e\") "
Jul 16 17:48:02.096864 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.096837    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e08602-8875-4c94-9f8b-07a239066a8e-kube-api-access-jttgt" (OuterVolumeSpecName: "kube-api-access-jttgt") pod "34e08602-8875-4c94-9f8b-07a239066a8e" (UID: "34e08602-8875-4c94-9f8b-07a239066a8e"). InnerVolumeSpecName "kube-api-access-jttgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:02.113941 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.113910    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e08602-8875-4c94-9f8b-07a239066a8e-model-cache" (OuterVolumeSpecName: "model-cache") pod "34e08602-8875-4c94-9f8b-07a239066a8e" (UID: "34e08602-8875-4c94-9f8b-07a239066a8e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:02.196155 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.196118    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34e08602-8875-4c94-9f8b-07a239066a8e-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:48:02.196155 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.196149    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttgt\" (UniqueName: \"kubernetes.io/projected/34e08602-8875-4c94-9f8b-07a239066a8e-kube-api-access-jttgt\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:48:02.855606 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.855577    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-4sjxq"
Jul 16 17:48:02.855784 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.855581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-4sjxq" event={"ID":"34e08602-8875-4c94-9f8b-07a239066a8e","Type":"ContainerDied","Data":"98f0374ba59973e5c16b09f868d5bae2ae4990f8e627c995478af37e2f9e1276"}
Jul 16 17:48:02.855784 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:02.855698    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f0374ba59973e5c16b09f868d5bae2ae4990f8e627c995478af37e2f9e1276"
Jul 16 17:48:25.712130 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.712092    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"]
Jul 16 17:48:25.712660 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.712640    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e08602-8875-4c94-9f8b-07a239066a8e" containerName="download-hf-model"
Jul 16 17:48:25.712740 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.712664    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e08602-8875-4c94-9f8b-07a239066a8e" containerName="download-hf-model"
Jul 16 17:48:25.712740 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.712695    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e08602-8875-4c94-9f8b-07a239066a8e" containerName="s3-init"
Jul 16 17:48:25.712740 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.712705    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e08602-8875-4c94-9f8b-07a239066a8e" containerName="s3-init"
Jul 16 17:48:25.712895 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.712801    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e08602-8875-4c94-9f8b-07a239066a8e" containerName="s3-init"
Jul 16 17:48:25.715944 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.715921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.720869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.720840    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:48:25.722285 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.722249    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\""
Jul 16 17:48:25.722500 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.722282    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:48:25.722626 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.722334    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:48:25.728673 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.728647    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"]
Jul 16 17:48:25.801339 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.801300    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.801517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.801383    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx7tl\" (UniqueName: \"kubernetes.io/projected/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-kube-api-access-mx7tl\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.801517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.801418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.801517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.801442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.801517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.801502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.801674 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.801579    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.902546 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx7tl\" (UniqueName: \"kubernetes.io/projected/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-kube-api-access-mx7tl\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.902730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.902730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902581    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.902730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.902730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.902730 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.903035 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.902998    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.903156 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.903049    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.903156 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.903071    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.904956 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.904936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.905194 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.905177    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:25.912962 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:25.912935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx7tl\" (UniqueName: \"kubernetes.io/projected/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-kube-api-access-mx7tl\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:26.027487 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:26.027390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:26.169346 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:26.169317    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"]
Jul 16 17:48:26.171162 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:48:26.171138    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eca6b20_0fcd_4a8f_9d23_55f20ddf081f.slice/crio-2fd126754d6e9a56025c42081ed7d2ff5d32559f8c3b85182454263f043bfd70 WatchSource:0}: Error finding container 2fd126754d6e9a56025c42081ed7d2ff5d32559f8c3b85182454263f043bfd70: Status 404 returned error can't find the container with id 2fd126754d6e9a56025c42081ed7d2ff5d32559f8c3b85182454263f043bfd70
Jul 16 17:48:26.981936 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:26.981896    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" event={"ID":"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f","Type":"ContainerStarted","Data":"2fd126754d6e9a56025c42081ed7d2ff5d32559f8c3b85182454263f043bfd70"}
Jul 16 17:48:27.988507 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:27.988468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" event={"ID":"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f","Type":"ContainerStarted","Data":"c40a4b37db347d34d95c2b78f05070cf5ea86b5cee8998e55fe7e9d55e5b24e8"}
Jul 16 17:48:28.012540 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:28.012488    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" podStartSLOduration=1.988156884 podStartE2EDuration="3.012471011s" totalImagesPullingTime="1.024314127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:28.010398265 +0000 UTC m=+829.393868233" watchObservedRunningTime="2026-07-16 17:48:28.012471011 +0000 UTC m=+829.395940945"
Jul 16 17:48:28.027622 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:28.027595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:36.028082 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:36.028043    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:36.040640 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:36.040612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:37.039763 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:37.039735    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:48:45.778517 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.778481    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"]
Jul 16 17:48:45.781113 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.781095    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.784005 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.783984    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:48:45.784862 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.784847    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 16 17:48:45.798528 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.798500    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"]
Jul 16 17:48:45.884177 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884138    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tls-certs\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.884383 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.884383 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fths\" (UniqueName: \"kubernetes.io/projected/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kube-api-access-6fths\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.884383 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.884524 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-dshm\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.884524 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-model-cache\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.884524 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.884439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-home\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.985731 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.985692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-dshm\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.985731 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.985732    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-model-cache\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.985994 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.985762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-home\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.985994 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.985823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tls-certs\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.985994 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.985870    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.986169 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.985994    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fths\" (UniqueName: \"kubernetes.io/projected/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kube-api-access-6fths\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.986169 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.986045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.986313 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.986242    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-home\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.986313 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.986305    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tmp-dir\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.986616 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.986323    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-model-cache\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.986616 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.986465    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.988089 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.988057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-dshm\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.988371 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.988354    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tls-certs\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:45.995675 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:45.995654    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fths\" (UniqueName: \"kubernetes.io/projected/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kube-api-access-6fths\") pod \"precise-prefix-cache-test-kserve-b4979bc47-dll9c\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:46.090972 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:46.090938    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:46.226095 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:46.226070    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"]
Jul 16 17:48:46.227598 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:48:46.227559    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ea54d3_2dbb_47a1_9c05_09396d7621fb.slice/crio-d91c1461d48f46d97970c224eff5cae9a8277557bd7bd8f99912e086b0f274be WatchSource:0}: Error finding container d91c1461d48f46d97970c224eff5cae9a8277557bd7bd8f99912e086b0f274be: Status 404 returned error can't find the container with id d91c1461d48f46d97970c224eff5cae9a8277557bd7bd8f99912e086b0f274be
Jul 16 17:48:47.094330 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:47.094258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" event={"ID":"04ea54d3-2dbb-47a1-9c05-09396d7621fb","Type":"ContainerStarted","Data":"61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc"}
Jul 16 17:48:47.094781 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:47.094381    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" event={"ID":"04ea54d3-2dbb-47a1-9c05-09396d7621fb","Type":"ContainerStarted","Data":"d91c1461d48f46d97970c224eff5cae9a8277557bd7bd8f99912e086b0f274be"}
Jul 16 17:48:48.947303 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:48.945919    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"]
Jul 16 17:48:50.112630 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:50.112594    2579 generic.go:356] "Generic (PLEG): container finished" podID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerID="61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc" exitCode=0
Jul 16 17:48:50.113038 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:50.112665    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" event={"ID":"04ea54d3-2dbb-47a1-9c05-09396d7621fb","Type":"ContainerDied","Data":"61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc"}
Jul 16 17:48:51.119651 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:51.119610    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" event={"ID":"04ea54d3-2dbb-47a1-9c05-09396d7621fb","Type":"ContainerStarted","Data":"29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca"}
Jul 16 17:48:51.157368 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:51.157314    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" podStartSLOduration=2.950979962 podStartE2EDuration="6.157296576s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.206316614s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:51.153311303 +0000 UTC m=+852.536781237" watchObservedRunningTime="2026-07-16 17:48:51.157296576 +0000 UTC m=+852.540766510"
Jul 16 17:48:52.091204 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:52.091156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:56.091749 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:56.091714    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:56.104323 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:56.104302    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:56.162831 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:56.162806    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:48:57.158842 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:57.158801    2579 generic.go:356] "Generic (PLEG): container finished" podID="67837974-612e-4f3a-9fc1-e70b10efec4e" containerID="d0119d25cd0d66f121243f0716ab9f1410ad051f682aaaf2f6430bd76d05e56e" exitCode=0
Jul 16 17:48:57.159384 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:57.158874    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" event={"ID":"67837974-612e-4f3a-9fc1-e70b10efec4e","Type":"ContainerDied","Data":"d0119d25cd0d66f121243f0716ab9f1410ad051f682aaaf2f6430bd76d05e56e"}
Jul 16 17:48:57.159716 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:57.159695    2579 scope.go:117] "RemoveContainer" containerID="d0119d25cd0d66f121243f0716ab9f1410ad051f682aaaf2f6430bd76d05e56e"
Jul 16 17:48:57.160758 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:57.160722    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 17:48:57.160845 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:57.160810    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 17:48:57.160886 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:57.160855    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 16 17:48:58.166682 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:58.166649    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" event={"ID":"67837974-612e-4f3a-9fc1-e70b10efec4e","Type":"ContainerStarted","Data":"11a27fa6f61396ee5a5fab1067bfe1d5c5d3e183dd5b5bc285b3cf8d659a8bd6"}
Jul 16 17:48:58.189354 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:58.189322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:48:58.189582 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:58.189560    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-qnxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 17:48:58.189664 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:58.189615    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" podUID="67837974-612e-4f3a-9fc1-e70b10efec4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 17:48:59.189172 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:59.189139    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-qnxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 17:48:59.189576 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:48:59.189192    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" podUID="67837974-612e-4f3a-9fc1-e70b10efec4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 17:49:00.189364 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:00.189327    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-qnxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 17:49:00.189763 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:00.189382    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" podUID="67837974-612e-4f3a-9fc1-e70b10efec4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 17:49:01.188776 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:01.188737    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-qnxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 17:49:01.188959 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:01.188799    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" podUID="67837974-612e-4f3a-9fc1-e70b10efec4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 17:49:02.189054 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:02.189020    2579 patch_prober.go:28] interesting pod/openshift-ai-inference-openshift-default-866675cbb8-qnxtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body=
Jul 16 17:49:02.189476 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:02.189073    2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj" podUID="67837974-612e-4f3a-9fc1-e70b10efec4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused"
Jul 16 17:49:03.491994 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:03.491962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:49:03.492498 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:03.492447    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:49:03.493320 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:03.493251    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-default-866675cbb8-qnxtj"
Jul 16 17:49:03.977549 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:03.977505    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" podUID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" containerName="main" containerID="cri-o://c40a4b37db347d34d95c2b78f05070cf5ea86b5cee8998e55fe7e9d55e5b24e8" gracePeriod=45
Jul 16 17:49:04.040095 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.040058    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" podUID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": dial tcp 10.133.0.45:8000: connect: connection refused"
Jul 16 17:49:04.202392 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.202352    2579 generic.go:356] "Generic (PLEG): container finished" podID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" containerID="c40a4b37db347d34d95c2b78f05070cf5ea86b5cee8998e55fe7e9d55e5b24e8" exitCode=0
Jul 16 17:49:04.202570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.202410    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" event={"ID":"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f","Type":"ContainerDied","Data":"c40a4b37db347d34d95c2b78f05070cf5ea86b5cee8998e55fe7e9d55e5b24e8"}
Jul 16 17:49:04.227819 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.227762    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:49:04.278817 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.278789    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tls-certs\") pod \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") "
Jul 16 17:49:04.278955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.278824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx7tl\" (UniqueName: \"kubernetes.io/projected/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-kube-api-access-mx7tl\") pod \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") "
Jul 16 17:49:04.278955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.278861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-model-cache\") pod \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") "
Jul 16 17:49:04.278955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.278879    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tmp-dir\") pod \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") "
Jul 16 17:49:04.278955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.278916    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-dshm\") pod \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") "
Jul 16 17:49:04.279182 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.278989    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-home\") pod \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\" (UID: \"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f\") "
Jul 16 17:49:04.279182 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.279129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-model-cache" (OuterVolumeSpecName: "model-cache") pod "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" (UID: "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:49:04.279334 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.279205    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" (UID: "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:49:04.279418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.279397    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:49:04.279485 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.279421    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:49:04.279630 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.279600    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-home" (OuterVolumeSpecName: "home") pod "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" (UID: "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:49:04.281153 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.281130    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" (UID: "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:49:04.281153 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.281145    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-dshm" (OuterVolumeSpecName: "dshm") pod "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" (UID: "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:49:04.281388 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.281370    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-kube-api-access-mx7tl" (OuterVolumeSpecName: "kube-api-access-mx7tl") pod "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" (UID: "0eca6b20-0fcd-4a8f-9d23-55f20ddf081f"). InnerVolumeSpecName "kube-api-access-mx7tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:49:04.380839 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.380799    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:49:04.380839 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.380832    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx7tl\" (UniqueName: \"kubernetes.io/projected/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-kube-api-access-mx7tl\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:49:04.380839 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.380841    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:49:04.380839 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:04.380849    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:49:05.209315 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:05.209282    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"
Jul 16 17:49:05.209315 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:05.209303    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq" event={"ID":"0eca6b20-0fcd-4a8f-9d23-55f20ddf081f","Type":"ContainerDied","Data":"2fd126754d6e9a56025c42081ed7d2ff5d32559f8c3b85182454263f043bfd70"}
Jul 16 17:49:05.209796 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:05.209372    2579 scope.go:117] "RemoveContainer" containerID="c40a4b37db347d34d95c2b78f05070cf5ea86b5cee8998e55fe7e9d55e5b24e8"
Jul 16 17:49:05.234147 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:05.234115    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"]
Jul 16 17:49:05.238982 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:05.238958    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-7c9c8cffc55rhwq"]
Jul 16 17:49:07.103518 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:07.103486    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" path="/var/lib/kubelet/pods/0eca6b20-0fcd-4a8f-9d23-55f20ddf081f/volumes"
Jul 16 17:49:17.615388 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.615353    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"]
Jul 16 17:49:17.615751 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.615712    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" containerName="main"
Jul 16 17:49:17.615751 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.615723    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" containerName="main"
Jul 16 17:49:17.615825 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.615814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eca6b20-0fcd-4a8f-9d23-55f20ddf081f" containerName="main"
Jul 16 17:49:17.618423 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.618404    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.621498 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.621480    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 16 17:49:17.630595 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.630575    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"]
Jul 16 17:49:17.690000 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.689966    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tmp-dir\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.690144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.690010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kserve-provision-location\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.690144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.690036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-home\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.690144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.690085    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tls-certs\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.690248 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.690174    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6slwv\" (UniqueName: \"kubernetes.io/projected/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kube-api-access-6slwv\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.690248 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.690199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-model-cache\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.690248 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.690229    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-dshm\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791064 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tmp-dir\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kserve-provision-location\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791094    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-home\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tls-certs\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6slwv\" (UniqueName: \"kubernetes.io/projected/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kube-api-access-6slwv\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-model-cache\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791220    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-dshm\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791637 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791529    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tmp-dir\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791637 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791555    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-home\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791747 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791664    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-model-cache\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.791747 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.791723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kserve-provision-location\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.793584 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.793561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-dshm\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.793791 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.793776    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tls-certs\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.809222 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.809198    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6slwv\" (UniqueName: \"kubernetes.io/projected/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kube-api-access-6slwv\") pod \"auth-enabled-test-kserve-798878c5c7-8z67f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:17.929907 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:17.929827    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:18.067171 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:18.067144    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"]
Jul 16 17:49:18.069422 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:49:18.069393    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eac9d39_11bc_4037_8d5e_95a9c3c98a2f.slice/crio-9fc6e3d6e51779e2cf941fd5fbb6b0434b4f3e97ac5033d96f64971d2a741586 WatchSource:0}: Error finding container 9fc6e3d6e51779e2cf941fd5fbb6b0434b4f3e97ac5033d96f64971d2a741586: Status 404 returned error can't find the container with id 9fc6e3d6e51779e2cf941fd5fbb6b0434b4f3e97ac5033d96f64971d2a741586
Jul 16 17:49:18.289791 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:18.289756    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" event={"ID":"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f","Type":"ContainerStarted","Data":"0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab"}
Jul 16 17:49:18.289929 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:18.289814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" event={"ID":"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f","Type":"ContainerStarted","Data":"9fc6e3d6e51779e2cf941fd5fbb6b0434b4f3e97ac5033d96f64971d2a741586"}
Jul 16 17:49:21.307216 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:21.307183    2579 generic.go:356] "Generic (PLEG): container finished" podID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerID="0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab" exitCode=0
Jul 16 17:49:21.307590 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:21.307239    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" event={"ID":"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f","Type":"ContainerDied","Data":"0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab"}
Jul 16 17:49:48.117700 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:48.117673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:49:48.118116 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:48.117833    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:49:48.507185 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:48.507150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" event={"ID":"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f","Type":"ContainerStarted","Data":"808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0"}
Jul 16 17:49:49.543808 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:49.543751    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podStartSLOduration=2.505521359 podStartE2EDuration="32.543732742s" totalImagesPullingTime="27.008696512s" totalInitContainerRuntime="3.029514871s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:49.540297876 +0000 UTC m=+910.923767811" watchObservedRunningTime="2026-07-16 17:49:49.543732742 +0000 UTC m=+910.927202725"
Jul 16 17:49:49.930687 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:49.930591    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:57.930859 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:57.930820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:49:57.932732 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:49:57.932704    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:50:07.930894 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:07.930841    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:50:17.931131 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:17.931082    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:50:27.930734 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:27.930684    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:50:34.459503 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:34.459461    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"]
Jul 16 17:50:37.931001 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:37.930960    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:50:47.930613 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:47.930564    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:50:49.480901 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.480841    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerName="main" containerID="cri-o://29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca" gracePeriod=45
Jul 16 17:50:49.730630 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.730603    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:50:49.843143 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.843108    2579 generic.go:356] "Generic (PLEG): container finished" podID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerID="29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca" exitCode=0
Jul 16 17:50:49.843369 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.843160    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"
Jul 16 17:50:49.843369 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.843192    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" event={"ID":"04ea54d3-2dbb-47a1-9c05-09396d7621fb","Type":"ContainerDied","Data":"29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca"}
Jul 16 17:50:49.843369 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.843252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c" event={"ID":"04ea54d3-2dbb-47a1-9c05-09396d7621fb","Type":"ContainerDied","Data":"d91c1461d48f46d97970c224eff5cae9a8277557bd7bd8f99912e086b0f274be"}
Jul 16 17:50:49.843369 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.843309    2579 scope.go:117] "RemoveContainer" containerID="29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca"
Jul 16 17:50:49.852136 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.852111    2579 scope.go:117] "RemoveContainer" containerID="61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc"
Jul 16 17:50:49.862308 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.862288    2579 scope.go:117] "RemoveContainer" containerID="29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca"
Jul 16 17:50:49.862569 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:50:49.862553    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca\": container with ID starting with 29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca not found: ID does not exist" containerID="29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca"
Jul 16 17:50:49.862623 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.862587    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca"} err="failed to get container status \"29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca\": rpc error: code = NotFound desc = could not find container \"29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca\": container with ID starting with 29defa614972589860483fc42a4f502a9730bbecee21f283d54b7e6a8c307cca not found: ID does not exist"
Jul 16 17:50:49.862623 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.862602    2579 scope.go:117] "RemoveContainer" containerID="61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc"
Jul 16 17:50:49.862862 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:50:49.862846    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc\": container with ID starting with 61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc not found: ID does not exist" containerID="61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc"
Jul 16 17:50:49.862906 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.862867    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc"} err="failed to get container status \"61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc\": rpc error: code = NotFound desc = could not find container \"61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc\": container with ID starting with 61b8434aff361734a670656cc86097a543237f9b2815c3c7f090357d9d32cacc not found: ID does not exist"
Jul 16 17:50:49.869174 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869156    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tls-certs\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869243 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fths\" (UniqueName: \"kubernetes.io/projected/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kube-api-access-6fths\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869327 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869250    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kserve-provision-location\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869327 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869289    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-home\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869344    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-dshm\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869365    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-model-cache\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869434 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869410    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tmp-dir\") pod \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\" (UID: \"04ea54d3-2dbb-47a1-9c05-09396d7621fb\") "
Jul 16 17:50:49.869728 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869612    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-home" (OuterVolumeSpecName: "home") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.869839 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869760    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.869890 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.869871    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.871743 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.871717    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:50:49.871743 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.871729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kube-api-access-6fths" (OuterVolumeSpecName: "kube-api-access-6fths") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "kube-api-access-6fths". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:50:49.871933 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.871870    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-dshm" (OuterVolumeSpecName: "dshm") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.924158 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.924113    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "04ea54d3-2dbb-47a1-9c05-09396d7621fb" (UID: "04ea54d3-2dbb-47a1-9c05-09396d7621fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:49.971003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.970968    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.971003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.970994    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fths\" (UniqueName: \"kubernetes.io/projected/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kube-api-access-6fths\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.971003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.971004    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.971003 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.971013    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.971382 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.971021    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.971382 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.971030    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:49.971382 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:49.971038    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04ea54d3-2dbb-47a1-9c05-09396d7621fb-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:50:50.185242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:50.185213    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"]
Jul 16 17:50:50.196043 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:50.191137    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-b4979bc47-dll9c"]
Jul 16 17:50:51.103380 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:51.103347    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" path="/var/lib/kubelet/pods/04ea54d3-2dbb-47a1-9c05-09396d7621fb/volumes"
Jul 16 17:50:57.931242 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:50:57.931200    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:51:07.931103 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:07.930992    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:51:17.930681 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:17.930637    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:51:27.931244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:27.931195    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Jul 16 17:51:37.940479 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:37.940441    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:51:37.948441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:37.948416    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:51:43.158609 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:43.158576    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"]
Jul 16 17:51:58.182278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:58.182203    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main" containerID="cri-o://808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0" gracePeriod=45
Jul 16 17:51:58.934683 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:58.934655    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:51:59.088363 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088318    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-model-cache\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.088570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088378    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tmp-dir\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.088570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088418    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-dshm\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.088570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088439    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-home\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.088570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088473    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kserve-provision-location\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.088570 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088500    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tls-certs\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.088860 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088675    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-model-cache" (OuterVolumeSpecName: "model-cache") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:59.088860 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.088845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6slwv\" (UniqueName: \"kubernetes.io/projected/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kube-api-access-6slwv\") pod \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\" (UID: \"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f\") "
Jul 16 17:51:59.089113 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.089084    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-home" (OuterVolumeSpecName: "home") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:59.089201 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.089143    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.090668 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.090644    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-dshm" (OuterVolumeSpecName: "dshm") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:59.091111 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.091081    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:51:59.091224 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.091090    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kube-api-access-6slwv" (OuterVolumeSpecName: "kube-api-access-6slwv") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "kube-api-access-6slwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:51:59.107222 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.107167    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:59.147988 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.147919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" (UID: "1eac9d39-11bc-4037-8d5e-95a9c3c98a2f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:51:59.189578 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.189542    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.189578 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.189574    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.189578 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.189584    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.190033 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.189594    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.190033 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.189604    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.190033 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.189613    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6slwv\" (UniqueName: \"kubernetes.io/projected/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f-kube-api-access-6slwv\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:51:59.230427 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.230386    2579 generic.go:356] "Generic (PLEG): container finished" podID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerID="808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0" exitCode=0
Jul 16 17:51:59.230595 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.230443    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"
Jul 16 17:51:59.230595 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.230475    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" event={"ID":"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f","Type":"ContainerDied","Data":"808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0"}
Jul 16 17:51:59.230595 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.230540    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f" event={"ID":"1eac9d39-11bc-4037-8d5e-95a9c3c98a2f","Type":"ContainerDied","Data":"9fc6e3d6e51779e2cf941fd5fbb6b0434b4f3e97ac5033d96f64971d2a741586"}
Jul 16 17:51:59.230595 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.230570    2579 scope.go:117] "RemoveContainer" containerID="808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0"
Jul 16 17:51:59.240278 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.240249    2579 scope.go:117] "RemoveContainer" containerID="0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab"
Jul 16 17:51:59.248676 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.248657    2579 scope.go:117] "RemoveContainer" containerID="808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0"
Jul 16 17:51:59.248939 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:51:59.248920    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0\": container with ID starting with 808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0 not found: ID does not exist" containerID="808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0"
Jul 16 17:51:59.249018 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.248952    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0"} err="failed to get container status \"808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0\": rpc error: code = NotFound desc = could not find container \"808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0\": container with ID starting with 808eeeac28b0305ebfd0cdbe29869a811a5f490f27e8bd75ac27af92db3da3c0 not found: ID does not exist"
Jul 16 17:51:59.249018 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.248977    2579 scope.go:117] "RemoveContainer" containerID="0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab"
Jul 16 17:51:59.249216 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:51:59.249200    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab\": container with ID starting with 0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab not found: ID does not exist" containerID="0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab"
Jul 16 17:51:59.249254 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.249221    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab"} err="failed to get container status \"0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab\": rpc error: code = NotFound desc = could not find container \"0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab\": container with ID starting with 0372ee4b617445b7dcc6a0ca65f5fa8408343f2737fad7752cefe8434a91e4ab not found: ID does not exist"
Jul 16 17:51:59.257713 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.257687    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"]
Jul 16 17:51:59.264825 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:51:59.264800    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-798878c5c7-8z67f"]
Jul 16 17:52:01.103728 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:01.103682    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" path="/var/lib/kubelet/pods/1eac9d39-11bc-4037-8d5e-95a9c3c98a2f/volumes"
Jul 16 17:52:18.885451 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.885415    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"]
Jul 16 17:52:18.885971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.885955    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerName="main"
Jul 16 17:52:18.886016 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.885974    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerName="main"
Jul 16 17:52:18.886016 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886001    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="storage-initializer"
Jul 16 17:52:18.886016 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886010    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="storage-initializer"
Jul 16 17:52:18.886115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886026    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerName="storage-initializer"
Jul 16 17:52:18.886115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886035    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerName="storage-initializer"
Jul 16 17:52:18.886115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886048    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main"
Jul 16 17:52:18.886115 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886056    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main"
Jul 16 17:52:18.886325 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886160    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ea54d3-2dbb-47a1-9c05-09396d7621fb" containerName="main"
Jul 16 17:52:18.886325 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.886172    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eac9d39-11bc-4037-8d5e-95a9c3c98a2f" containerName="main"
Jul 16 17:52:18.889901 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.889880    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.892864 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.892842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:52:18.893005 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.892969    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:52:18.893076 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.892972    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:52:18.893076 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.893032    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 16 17:52:18.893361 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.893333    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:52:18.900958 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.900936    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"]
Jul 16 17:52:18.971124 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-home\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.971322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971126    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-model-cache\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.971322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tmp-dir\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.971322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971225    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-dshm\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.971322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971289    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.971473 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971345    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsz4c\" (UniqueName: \"kubernetes.io/projected/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kube-api-access-jsz4c\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:18.971473 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:18.971365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tls-certs\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072580 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072546    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-home\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072580 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-model-cache\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tmp-dir\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-dshm\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072662    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072725    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsz4c\" (UniqueName: \"kubernetes.io/projected/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kube-api-access-jsz4c\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.072828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.072752    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tls-certs\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.073073 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.073054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tmp-dir\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.073137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.073090    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-home\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.073137 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.073106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-model-cache\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.073259 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.073170    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.075045 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.075013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-dshm\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.075217 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.075199    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tls-certs\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.083797 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.083772    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsz4c\" (UniqueName: \"kubernetes.io/projected/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kube-api-access-jsz4c\") pod \"auth-invalid-token-test-kserve-69956d8989-pnbmz\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.176870 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.176783    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"]
Jul 16 17:52:19.180631 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.180609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.183644 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.183621    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-4wknw\""
Jul 16 17:52:19.195605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.195580    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"]
Jul 16 17:52:19.202043 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.202017    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:19.275728 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.275695    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.275974 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.275742    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.275974 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.275765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx2px\" (UniqueName: \"kubernetes.io/projected/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kube-api-access-xx2px\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.275974 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.275803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.367208 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.367179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"]
Jul 16 17:52:19.367781 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:52:19.367753    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f24c1c_2819_4ede_b8e9_9e33f3435b17.slice/crio-1109de3e040e0c42a55bd4410f3ab3f3e91b4e138bc5834aecf75ca05e66fccf WatchSource:0}: Error finding container 1109de3e040e0c42a55bd4410f3ab3f3e91b4e138bc5834aecf75ca05e66fccf: Status 404 returned error can't find the container with id 1109de3e040e0c42a55bd4410f3ab3f3e91b4e138bc5834aecf75ca05e66fccf
Jul 16 17:52:19.376647 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.376623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx2px\" (UniqueName: \"kubernetes.io/projected/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kube-api-access-xx2px\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.376714 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.376698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.376849 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.376833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.376893 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.376875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.377096 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.377079    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.377190 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.377171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.379186 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.379157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.389063 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.389035    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx2px\" (UniqueName: \"kubernetes.io/projected/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kube-api-access-xx2px\") pod \"auth-invalid-token-test-kserve-router-scheduler-5649968468lk857\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.490506 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.490413    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:19.631220 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:19.631195    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"]
Jul 16 17:52:19.632729 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:52:19.632702    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfd6bbb_f2f4_460e_a7b7_380c721d8875.slice/crio-6e186d30ad8f0c5fabbeca90f9e6b4eea75942d311ae1dde645944c455e98ab9 WatchSource:0}: Error finding container 6e186d30ad8f0c5fabbeca90f9e6b4eea75942d311ae1dde645944c455e98ab9: Status 404 returned error can't find the container with id 6e186d30ad8f0c5fabbeca90f9e6b4eea75942d311ae1dde645944c455e98ab9
Jul 16 17:52:20.343971 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:20.343910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" event={"ID":"e7f24c1c-2819-4ede-b8e9-9e33f3435b17","Type":"ContainerStarted","Data":"2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a"}
Jul 16 17:52:20.344441 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:20.344030    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" event={"ID":"e7f24c1c-2819-4ede-b8e9-9e33f3435b17","Type":"ContainerStarted","Data":"1109de3e040e0c42a55bd4410f3ab3f3e91b4e138bc5834aecf75ca05e66fccf"}
Jul 16 17:52:20.345840 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:20.345805    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" event={"ID":"cdfd6bbb-f2f4-460e-a7b7-380c721d8875","Type":"ContainerStarted","Data":"da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2"}
Jul 16 17:52:20.345960 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:20.345866    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" event={"ID":"cdfd6bbb-f2f4-460e-a7b7-380c721d8875","Type":"ContainerStarted","Data":"6e186d30ad8f0c5fabbeca90f9e6b4eea75942d311ae1dde645944c455e98ab9"}
Jul 16 17:52:21.355205 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:21.355164    2579 generic.go:356] "Generic (PLEG): container finished" podID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerID="da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2" exitCode=0
Jul 16 17:52:21.355782 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:21.355254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" event={"ID":"cdfd6bbb-f2f4-460e-a7b7-380c721d8875","Type":"ContainerDied","Data":"da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2"}
Jul 16 17:52:21.356922 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:21.356801    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:52:23.370828 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:23.370796    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerID="2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a" exitCode=0
Jul 16 17:52:23.371299 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:23.370839    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" event={"ID":"e7f24c1c-2819-4ede-b8e9-9e33f3435b17","Type":"ContainerDied","Data":"2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a"}
Jul 16 17:52:24.378853 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:24.378751    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" event={"ID":"e7f24c1c-2819-4ede-b8e9-9e33f3435b17","Type":"ContainerStarted","Data":"6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8"}
Jul 16 17:52:24.380931 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:24.380907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" event={"ID":"cdfd6bbb-f2f4-460e-a7b7-380c721d8875","Type":"ContainerStarted","Data":"7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7"}
Jul 16 17:52:24.381084 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:24.380979    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:24.413195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:24.413123    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podStartSLOduration=3.331605096 podStartE2EDuration="6.413102746s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.08149765s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:24.411749876 +0000 UTC m=+1065.795219812" watchObservedRunningTime="2026-07-16 17:52:24.413102746 +0000 UTC m=+1065.796572682"
Jul 16 17:52:24.436556 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:24.436503    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" podStartSLOduration=1.840486994 podStartE2EDuration="5.436486105s" totalImagesPullingTime="2.766260474s" totalInitContainerRuntime="829.738637ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:52:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:52:24.433772761 +0000 UTC m=+1065.817242696" watchObservedRunningTime="2026-07-16 17:52:24.436486105 +0000 UTC m=+1065.819956038"
Jul 16 17:52:25.202880 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:25.202839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:29.203228 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:29.203183    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:52:29.204940 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:29.204907    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:52:39.202761 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:39.202654    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:52:49.202844 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:49.202797    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:52:55.387960 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:55.387929    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:52:59.203381 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:52:59.203338    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:53:09.203342 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:53:09.203295    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:53:19.203424 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:53:19.203324    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:53:29.202544 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:53:29.202497    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:53:39.202921 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:53:39.202868    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:53:49.202767 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:53:49.202718    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:53:59.202589 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:53:59.202537    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:54:09.202726 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:09.202673    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:54:19.213689 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:19.213606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:54:19.221489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:19.221463    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:54:48.162716 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:48.162683    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:54:48.163863 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:48.163839    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:54:49.752602 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:49.752567    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"]
Jul 16 17:54:49.754887 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:49.754859    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"]
Jul 16 17:54:55.387807 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:54:55.387768    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:55:04.775167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:04.775112    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="main" containerID="cri-o://7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7" gracePeriod=45
Jul 16 17:55:04.776128 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:04.776098    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" containerID="cri-o://6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8" gracePeriod=45
Jul 16 17:55:05.031934 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.031873    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:55:05.211527 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.211483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kserve-provision-location\") pod \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") "
Jul 16 17:55:05.211527 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.211532    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx2px\" (UniqueName: \"kubernetes.io/projected/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kube-api-access-xx2px\") pod \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") "
Jul 16 17:55:05.211790 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.211620    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tokenizer-uds\") pod \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") "
Jul 16 17:55:05.211790 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.211670    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tls-certs\") pod \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\" (UID: \"cdfd6bbb-f2f4-460e-a7b7-380c721d8875\") "
Jul 16 17:55:05.212093 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.211899    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "cdfd6bbb-f2f4-460e-a7b7-380c721d8875" (UID: "cdfd6bbb-f2f4-460e-a7b7-380c721d8875"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.212093 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.212063    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tokenizer-uds\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.212348 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.212326    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdfd6bbb-f2f4-460e-a7b7-380c721d8875" (UID: "cdfd6bbb-f2f4-460e-a7b7-380c721d8875"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.213847 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.213824    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kube-api-access-xx2px" (OuterVolumeSpecName: "kube-api-access-xx2px") pod "cdfd6bbb-f2f4-460e-a7b7-380c721d8875" (UID: "cdfd6bbb-f2f4-460e-a7b7-380c721d8875"). InnerVolumeSpecName "kube-api-access-xx2px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:55:05.213847 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.213826    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdfd6bbb-f2f4-460e-a7b7-380c721d8875" (UID: "cdfd6bbb-f2f4-460e-a7b7-380c721d8875"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:55:05.214235 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.214195    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 16 17:55:05.287897 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.287808    2579 generic.go:356] "Generic (PLEG): container finished" podID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerID="7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7" exitCode=0
Jul 16 17:55:05.287897 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.287864    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"
Jul 16 17:55:05.288098 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.287898    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" event={"ID":"cdfd6bbb-f2f4-460e-a7b7-380c721d8875","Type":"ContainerDied","Data":"7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7"}
Jul 16 17:55:05.288098 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.287964    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857" event={"ID":"cdfd6bbb-f2f4-460e-a7b7-380c721d8875","Type":"ContainerDied","Data":"6e186d30ad8f0c5fabbeca90f9e6b4eea75942d311ae1dde645944c455e98ab9"}
Jul 16 17:55:05.288098 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.287991    2579 scope.go:117] "RemoveContainer" containerID="7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7"
Jul 16 17:55:05.298165 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.298147    2579 scope.go:117] "RemoveContainer" containerID="da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2"
Jul 16 17:55:05.307170 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.307150    2579 scope.go:117] "RemoveContainer" containerID="7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7"
Jul 16 17:55:05.307500 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:55:05.307479    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7\": container with ID starting with 7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7 not found: ID does not exist" containerID="7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7"
Jul 16 17:55:05.307566 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.307509    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7"} err="failed to get container status \"7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7\": rpc error: code = NotFound desc = could not find container \"7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7\": container with ID starting with 7d43dea7cbf19481162720b1bdf544f9b54fbb328ff978990a8629c1d882cbe7 not found: ID does not exist"
Jul 16 17:55:05.307566 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.307526    2579 scope.go:117] "RemoveContainer" containerID="da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2"
Jul 16 17:55:05.307769 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:55:05.307749    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2\": container with ID starting with da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2 not found: ID does not exist" containerID="da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2"
Jul 16 17:55:05.307816 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.307775    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2"} err="failed to get container status \"da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2\": rpc error: code = NotFound desc = could not find container \"da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2\": container with ID starting with da29fda8379f77230b53c1e66ed6703cceeb4e0dd585cfb031a9b6fe4a1533d2 not found: ID does not exist"
Jul 16 17:55:05.313321 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.313301    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.313396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.313323    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.313396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.313333    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx2px\" (UniqueName: \"kubernetes.io/projected/cdfd6bbb-f2f4-460e-a7b7-380c721d8875-kube-api-access-xx2px\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.319753 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.319731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"]
Jul 16 17:55:05.327754 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.327734    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5649968468lk857"]
Jul 16 17:55:05.525364 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.525335    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:55:05.715751 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.715718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kserve-provision-location\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.715970 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.715775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tls-certs\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.715970 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.715806    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-model-cache\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.715970 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.715824    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsz4c\" (UniqueName: \"kubernetes.io/projected/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kube-api-access-jsz4c\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.715970 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.715870    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tmp-dir\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.716195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.716020    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-home\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.716195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.716064    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-dshm\") pod \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\" (UID: \"e7f24c1c-2819-4ede-b8e9-9e33f3435b17\") "
Jul 16 17:55:05.716195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.716156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.716501 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.716455    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.716680 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.716655    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-home" (OuterVolumeSpecName: "home") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.718185 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.718163    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-dshm" (OuterVolumeSpecName: "dshm") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.718481 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.718456    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kube-api-access-jsz4c" (OuterVolumeSpecName: "kube-api-access-jsz4c") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "kube-api-access-jsz4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:55:05.718591 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.718493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:55:05.727183 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.727160    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.777819 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.777756    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7f24c1c-2819-4ede-b8e9-9e33f3435b17" (UID: "e7f24c1c-2819-4ede-b8e9-9e33f3435b17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:55:05.817233 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.817201    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.817233 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.817228    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.817233 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.817236    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.817476 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.817244    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.817476 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.817253    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:05.817476 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:05.817261    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsz4c\" (UniqueName: \"kubernetes.io/projected/e7f24c1c-2819-4ede-b8e9-9e33f3435b17-kube-api-access-jsz4c\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:55:06.295476 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.295440    2579 generic.go:356] "Generic (PLEG): container finished" podID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerID="6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8" exitCode=0
Jul 16 17:55:06.295647 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.295490    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"
Jul 16 17:55:06.295647 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.295524    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" event={"ID":"e7f24c1c-2819-4ede-b8e9-9e33f3435b17","Type":"ContainerDied","Data":"6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8"}
Jul 16 17:55:06.295647 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.295575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz" event={"ID":"e7f24c1c-2819-4ede-b8e9-9e33f3435b17","Type":"ContainerDied","Data":"1109de3e040e0c42a55bd4410f3ab3f3e91b4e138bc5834aecf75ca05e66fccf"}
Jul 16 17:55:06.295647 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.295600    2579 scope.go:117] "RemoveContainer" containerID="6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8"
Jul 16 17:55:06.305199 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.305178    2579 scope.go:117] "RemoveContainer" containerID="2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a"
Jul 16 17:55:06.313633 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.313619    2579 scope.go:117] "RemoveContainer" containerID="6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8"
Jul 16 17:55:06.313884 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:55:06.313856    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8\": container with ID starting with 6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8 not found: ID does not exist" containerID="6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8"
Jul 16 17:55:06.313933 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.313892    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8"} err="failed to get container status \"6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8\": rpc error: code = NotFound desc = could not find container \"6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8\": container with ID starting with 6f1e288e7699b55dbe44ddea00f766bf2dd477e39c0532129624b1084c5a76e8 not found: ID does not exist"
Jul 16 17:55:06.313933 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.313907    2579 scope.go:117] "RemoveContainer" containerID="2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a"
Jul 16 17:55:06.314105 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:55:06.314090    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a\": container with ID starting with 2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a not found: ID does not exist" containerID="2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a"
Jul 16 17:55:06.314142 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.314118    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a"} err="failed to get container status \"2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a\": rpc error: code = NotFound desc = could not find container \"2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a\": container with ID starting with 2373d94e518172ddd8880e85ec4849125eec4ba4df06e6b1bb22a3f5da94434a not found: ID does not exist"
Jul 16 17:55:06.323955 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.323935    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"]
Jul 16 17:55:06.329471 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:06.329450    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69956d8989-pnbmz"]
Jul 16 17:55:07.103766 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:07.103734    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" path="/var/lib/kubelet/pods/cdfd6bbb-f2f4-460e-a7b7-380c721d8875/volumes"
Jul 16 17:55:07.104167 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:07.104154    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" path="/var/lib/kubelet/pods/e7f24c1c-2819-4ede-b8e9-9e33f3435b17/volumes"
Jul 16 17:55:15.818461 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"]
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818788    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818799    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818810    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="storage-initializer"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818816    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="storage-initializer"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818842    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="storage-initializer"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818848    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="storage-initializer"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818854    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="main"
Jul 16 17:55:15.818867 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818859    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="main"
Jul 16 17:55:15.819120 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818912    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdfd6bbb-f2f4-460e-a7b7-380c721d8875" containerName="main"
Jul 16 17:55:15.819120 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.818923    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7f24c1c-2819-4ede-b8e9-9e33f3435b17" containerName="main"
Jul 16 17:55:15.822104 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.822086    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.825171 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.825138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:55:15.825171 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.825149    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:55:15.825673 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.825657    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 16 17:55:15.826501 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.826481    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:55:15.826960 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.826944    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:55:15.857710 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.857666    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"]
Jul 16 17:55:15.901847 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.901810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-kserve-provision-location\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.901847 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.901858    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29f8434-84fe-426d-8529-0508870a20eb-tls-certs\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.902077 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.901916    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-home\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.902077 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.901962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-dshm\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.902077 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.901991    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tqwp\" (UniqueName: \"kubernetes.io/projected/f29f8434-84fe-426d-8529-0508870a20eb-kube-api-access-4tqwp\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.902077 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.902017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-model-cache\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:15.902212 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:15.902096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-tmp-dir\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003433 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003397    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-home\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003610 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003448    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-dshm\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003610 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003568    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tqwp\" (UniqueName: \"kubernetes.io/projected/f29f8434-84fe-426d-8529-0508870a20eb-kube-api-access-4tqwp\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003727 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-model-cache\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003727 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003688    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-tmp-dir\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003916 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003740    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-kserve-provision-location\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.003916 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003777    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29f8434-84fe-426d-8529-0508870a20eb-tls-certs\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.004029 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.003936    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-home\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.004029 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.004003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-model-cache\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.004109 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.004085    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-kserve-provision-location\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.004144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.004125    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-tmp-dir\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.005760 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.005744    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-dshm\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.006327 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.006308    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29f8434-84fe-426d-8529-0508870a20eb-tls-certs\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.014541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.014512    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tqwp\" (UniqueName: \"kubernetes.io/projected/f29f8434-84fe-426d-8529-0508870a20eb-kube-api-access-4tqwp\") pod \"auth-disabled-test-kserve-869d748779-67hnj\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.096407 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.096257    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"]
Jul 16 17:55:16.101008 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.100981    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.103814 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.103792    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-dwq5d\""
Jul 16 17:55:16.110535 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.110513    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"]
Jul 16 17:55:16.132679 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.132646    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:16.205295 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.205236    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.205472 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.205323    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkb2\" (UniqueName: \"kubernetes.io/projected/7c54b169-1b7a-4f69-b014-31dab894a2de-kube-api-access-tvkb2\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.205472 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.205429    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.205472 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.205455    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b169-1b7a-4f69-b014-31dab894a2de-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.279717 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.279589    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"]
Jul 16 17:55:16.281976 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:55:16.281947    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29f8434_84fe_426d_8529_0508870a20eb.slice/crio-4b703411d71359503353a307e6c84c0ed00840b4503bd0fd8d30a5f9669cc0bf WatchSource:0}: Error finding container 4b703411d71359503353a307e6c84c0ed00840b4503bd0fd8d30a5f9669cc0bf: Status 404 returned error can't find the container with id 4b703411d71359503353a307e6c84c0ed00840b4503bd0fd8d30a5f9669cc0bf
Jul 16 17:55:16.305990 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.305961    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkb2\" (UniqueName: \"kubernetes.io/projected/7c54b169-1b7a-4f69-b014-31dab894a2de-kube-api-access-tvkb2\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.306106 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.306089    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.306149 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.306121    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b169-1b7a-4f69-b014-31dab894a2de-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.306149 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.306143    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.306447 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.306429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.306489 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.306453    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.308542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.308525    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b169-1b7a-4f69-b014-31dab894a2de-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.316443 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.316423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkb2\" (UniqueName: \"kubernetes.io/projected/7c54b169-1b7a-4f69-b014-31dab894a2de-kube-api-access-tvkb2\") pod \"auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.352026 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.351989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" event={"ID":"f29f8434-84fe-426d-8529-0508870a20eb","Type":"ContainerStarted","Data":"4b703411d71359503353a307e6c84c0ed00840b4503bd0fd8d30a5f9669cc0bf"}
Jul 16 17:55:16.411919 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.411876    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:16.561424 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:16.561388    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"]
Jul 16 17:55:16.562222 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:55:16.562199    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c54b169_1b7a_4f69_b014_31dab894a2de.slice/crio-bb418056698b08b4391e594663e62235781c085d0401444b76f756d94d14451c WatchSource:0}: Error finding container bb418056698b08b4391e594663e62235781c085d0401444b76f756d94d14451c: Status 404 returned error can't find the container with id bb418056698b08b4391e594663e62235781c085d0401444b76f756d94d14451c
Jul 16 17:55:17.363536 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:17.363461    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" event={"ID":"7c54b169-1b7a-4f69-b014-31dab894a2de","Type":"ContainerStarted","Data":"9e0687b62e81c33571ffb2de60ab7e2d305454128611141bebba8440321c685b"}
Jul 16 17:55:17.363536 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:17.363538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" event={"ID":"7c54b169-1b7a-4f69-b014-31dab894a2de","Type":"ContainerStarted","Data":"bb418056698b08b4391e594663e62235781c085d0401444b76f756d94d14451c"}
Jul 16 17:55:17.365722 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:17.365693    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" event={"ID":"f29f8434-84fe-426d-8529-0508870a20eb","Type":"ContainerStarted","Data":"047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a"}
Jul 16 17:55:18.377637 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:18.377593    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerID="9e0687b62e81c33571ffb2de60ab7e2d305454128611141bebba8440321c685b" exitCode=0
Jul 16 17:55:18.378322 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:18.377676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" event={"ID":"7c54b169-1b7a-4f69-b014-31dab894a2de","Type":"ContainerDied","Data":"9e0687b62e81c33571ffb2de60ab7e2d305454128611141bebba8440321c685b"}
Jul 16 17:55:19.384504 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:19.384464    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" event={"ID":"7c54b169-1b7a-4f69-b014-31dab894a2de","Type":"ContainerStarted","Data":"fd14e4795240bd9890631a916eaeeed8b2e03958c97ee1bd6c4d1ab5c7ce82af"}
Jul 16 17:55:19.385082 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:19.384626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:19.418067 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:19.418008    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" podStartSLOduration=2.62696676 podStartE2EDuration="3.417989116s" totalImagesPullingTime="0s" totalInitContainerRuntime="791.022356ms" isStatefulPod=false podCreationTimestamp="2026-07-16 17:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:55:19.416285481 +0000 UTC m=+1240.799755407" watchObservedRunningTime="2026-07-16 17:55:19.417989116 +0000 UTC m=+1240.801459049"
Jul 16 17:55:20.391554 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:20.391515    2579 generic.go:356] "Generic (PLEG): container finished" podID="f29f8434-84fe-426d-8529-0508870a20eb" containerID="047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a" exitCode=0
Jul 16 17:55:20.392094 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:20.391587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" event={"ID":"f29f8434-84fe-426d-8529-0508870a20eb","Type":"ContainerDied","Data":"047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a"}
Jul 16 17:55:21.400630 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:21.400592    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" event={"ID":"f29f8434-84fe-426d-8529-0508870a20eb","Type":"ContainerStarted","Data":"6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0"}
Jul 16 17:55:21.433700 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:21.433634    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podStartSLOduration=3.182607652 podStartE2EDuration="6.433618487s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.251010835s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:55:21.431237879 +0000 UTC m=+1242.814707819" watchObservedRunningTime="2026-07-16 17:55:21.433618487 +0000 UTC m=+1242.817088421"
Jul 16 17:55:22.132829 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:22.132789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:26.133654 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:26.133616    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:55:26.135392 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:26.135361    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:55:36.134151 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:36.134053    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:55:46.133985 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:46.133943    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:55:50.394368 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:50.394335    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:55:56.133524 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:55:56.133486    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:56:06.133583 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:56:06.133539    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:56:16.133859 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:56:16.133806    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:56:26.133949 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:56:26.133896    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:56:36.133392 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:56:36.133341    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:56:46.133509 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:56:46.133457    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:56:56.133467 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:56:56.133416    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:57:06.134147 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:06.134055    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:57:16.142896 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:16.142865    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:57:16.151037 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:16.151009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:57:22.920256 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.920219    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"]
Jul 16 17:57:22.925164 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.925142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.928210 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.928185    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 16 17:57:22.928362 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.928188    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:57:22.937040 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.937016    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"]
Jul 16 17:57:22.990195 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990199    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990310    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990396 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990373    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990565 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990407    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990565 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990421    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:22.990565 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:22.990483    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw64h\" (UniqueName: \"kubernetes.io/projected/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kube-api-access-pw64h\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091332 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091332 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091392    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091415    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091475    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091586 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091535    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw64h\" (UniqueName: \"kubernetes.io/projected/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kube-api-access-pw64h\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091903 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091774    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.091903 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.091851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.092162 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.092138    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.092245 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.092146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.092245 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.092227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.093807 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.093781    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.094497 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.094475    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.103163 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.103135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw64h\" (UniqueName: \"kubernetes.io/projected/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kube-api-access-pw64h\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.236311 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.236196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:23.394056 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:23.394026    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"]
Jul 16 17:57:23.396202 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:57:23.396173    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf582090b_fdb6_4c17_a8bd_e0c3bdbed948.slice/crio-47dd726945269961611964c1ba6fbe254d9f8c466a7c4d5f5837fcf9b344ce45 WatchSource:0}: Error finding container 47dd726945269961611964c1ba6fbe254d9f8c466a7c4d5f5837fcf9b344ce45: Status 404 returned error can't find the container with id 47dd726945269961611964c1ba6fbe254d9f8c466a7c4d5f5837fcf9b344ce45
Jul 16 17:57:24.115981 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:24.115939    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" event={"ID":"f582090b-fdb6-4c17-a8bd-e0c3bdbed948","Type":"ContainerStarted","Data":"415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43"}
Jul 16 17:57:24.116439 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:24.116008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" event={"ID":"f582090b-fdb6-4c17-a8bd-e0c3bdbed948","Type":"ContainerStarted","Data":"47dd726945269961611964c1ba6fbe254d9f8c466a7c4d5f5837fcf9b344ce45"}
Jul 16 17:57:26.064589 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:26.064520    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"]
Jul 16 17:57:26.068240 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:26.068200    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"]
Jul 16 17:57:27.137848 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:27.137810    2579 generic.go:356] "Generic (PLEG): container finished" podID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerID="415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43" exitCode=0
Jul 16 17:57:27.138258 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:27.137881    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" event={"ID":"f582090b-fdb6-4c17-a8bd-e0c3bdbed948","Type":"ContainerDied","Data":"415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43"}
Jul 16 17:57:28.147357 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:28.147309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" event={"ID":"f582090b-fdb6-4c17-a8bd-e0c3bdbed948","Type":"ContainerStarted","Data":"6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217"}
Jul 16 17:57:28.176078 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:28.176007    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podStartSLOduration=2.6778937149999997 podStartE2EDuration="6.175983471s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.498089756s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:57:28.174705658 +0000 UTC m=+1369.558175594" watchObservedRunningTime="2026-07-16 17:57:28.175983471 +0000 UTC m=+1369.559453406"
Jul 16 17:57:28.237199 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:28.237156    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:30.393523 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:30.393477    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:57:33.236853 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:33.236817    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:57:33.238464 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:33.238436    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:57:40.394173 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:40.394126    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 17:57:41.095605 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.095554    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" containerID="cri-o://6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0" gracePeriod=45
Jul 16 17:57:41.104300 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.104247    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="main" containerID="cri-o://fd14e4795240bd9890631a916eaeeed8b2e03958c97ee1bd6c4d1ab5c7ce82af" gracePeriod=45
Jul 16 17:57:41.143964 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.143918    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Jul 16 17:57:41.230463 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.230430    2579 generic.go:356] "Generic (PLEG): container finished" podID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerID="fd14e4795240bd9890631a916eaeeed8b2e03958c97ee1bd6c4d1ab5c7ce82af" exitCode=0
Jul 16 17:57:41.230593 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.230492    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" event={"ID":"7c54b169-1b7a-4f69-b014-31dab894a2de","Type":"ContainerDied","Data":"fd14e4795240bd9890631a916eaeeed8b2e03958c97ee1bd6c4d1ab5c7ce82af"}
Jul 16 17:57:41.383206 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.383174    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:57:41.475157 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.475127    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-tokenizer-uds\") pod \"7c54b169-1b7a-4f69-b014-31dab894a2de\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") "
Jul 16 17:57:41.475669 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.475251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvkb2\" (UniqueName: \"kubernetes.io/projected/7c54b169-1b7a-4f69-b014-31dab894a2de-kube-api-access-tvkb2\") pod \"7c54b169-1b7a-4f69-b014-31dab894a2de\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") "
Jul 16 17:57:41.475669 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.475313    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-kserve-provision-location\") pod \"7c54b169-1b7a-4f69-b014-31dab894a2de\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") "
Jul 16 17:57:41.475669 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.475371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b169-1b7a-4f69-b014-31dab894a2de-tls-certs\") pod \"7c54b169-1b7a-4f69-b014-31dab894a2de\" (UID: \"7c54b169-1b7a-4f69-b014-31dab894a2de\") "
Jul 16 17:57:41.475669 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.475495    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "7c54b169-1b7a-4f69-b014-31dab894a2de" (UID: "7c54b169-1b7a-4f69-b014-31dab894a2de"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:41.475892 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.475712    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-tokenizer-uds\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:41.476151 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.476112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7c54b169-1b7a-4f69-b014-31dab894a2de" (UID: "7c54b169-1b7a-4f69-b014-31dab894a2de"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:41.477471 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.477441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c54b169-1b7a-4f69-b014-31dab894a2de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c54b169-1b7a-4f69-b014-31dab894a2de" (UID: "7c54b169-1b7a-4f69-b014-31dab894a2de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:57:41.477662 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.477640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c54b169-1b7a-4f69-b014-31dab894a2de-kube-api-access-tvkb2" (OuterVolumeSpecName: "kube-api-access-tvkb2") pod "7c54b169-1b7a-4f69-b014-31dab894a2de" (UID: "7c54b169-1b7a-4f69-b014-31dab894a2de"). InnerVolumeSpecName "kube-api-access-tvkb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:57:41.576840 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.576797    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkb2\" (UniqueName: \"kubernetes.io/projected/7c54b169-1b7a-4f69-b014-31dab894a2de-kube-api-access-tvkb2\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:41.576840 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.576837    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7c54b169-1b7a-4f69-b014-31dab894a2de-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:41.577117 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.576851    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c54b169-1b7a-4f69-b014-31dab894a2de-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:41.853811 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.853784    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:57:41.981031 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.980930    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-home\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981031 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981008    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tqwp\" (UniqueName: \"kubernetes.io/projected/f29f8434-84fe-426d-8529-0508870a20eb-kube-api-access-4tqwp\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981031 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981035    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-model-cache\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-dshm\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981112    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-kserve-provision-location\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29f8434-84fe-426d-8529-0508870a20eb-tls-certs\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981351 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981190    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-tmp-dir\") pod \"f29f8434-84fe-426d-8529-0508870a20eb\" (UID: \"f29f8434-84fe-426d-8529-0508870a20eb\") "
Jul 16 17:57:41.981582 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981426    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:41.981639 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981614    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-home" (OuterVolumeSpecName: "home") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:41.981942 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981916    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:41.982051 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.981946    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:41.983508 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.983476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29f8434-84fe-426d-8529-0508870a20eb-kube-api-access-4tqwp" (OuterVolumeSpecName: "kube-api-access-4tqwp") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "kube-api-access-4tqwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:57:41.983607 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.983534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-dshm" (OuterVolumeSpecName: "dshm") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:41.983998 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.983977    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29f8434-84fe-426d-8529-0508870a20eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:57:41.999261 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:41.999219    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:42.049338 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.049290    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f29f8434-84fe-426d-8529-0508870a20eb" (UID: "f29f8434-84fe-426d-8529-0508870a20eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:57:42.083049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.083001    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:42.083049 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.083051    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29f8434-84fe-426d-8529-0508870a20eb-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:42.083380 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.083071    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:42.083380 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.083085    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tqwp\" (UniqueName: \"kubernetes.io/projected/f29f8434-84fe-426d-8529-0508870a20eb-kube-api-access-4tqwp\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:42.083380 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.083097    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29f8434-84fe-426d-8529-0508870a20eb-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:57:42.241643 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.241545    2579 generic.go:356] "Generic (PLEG): container finished" podID="f29f8434-84fe-426d-8529-0508870a20eb" containerID="6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0" exitCode=0
Jul 16 17:57:42.241643 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.241603    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"
Jul 16 17:57:42.241643 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.241629    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" event={"ID":"f29f8434-84fe-426d-8529-0508870a20eb","Type":"ContainerDied","Data":"6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0"}
Jul 16 17:57:42.241942 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.241688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj" event={"ID":"f29f8434-84fe-426d-8529-0508870a20eb","Type":"ContainerDied","Data":"4b703411d71359503353a307e6c84c0ed00840b4503bd0fd8d30a5f9669cc0bf"}
Jul 16 17:57:42.241942 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.241721    2579 scope.go:117] "RemoveContainer" containerID="6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0"
Jul 16 17:57:42.243593 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.243563    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"
Jul 16 17:57:42.243593 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.243578    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7" event={"ID":"7c54b169-1b7a-4f69-b014-31dab894a2de","Type":"ContainerDied","Data":"bb418056698b08b4391e594663e62235781c085d0401444b76f756d94d14451c"}
Jul 16 17:57:42.253150 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.253132    2579 scope.go:117] "RemoveContainer" containerID="047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a"
Jul 16 17:57:42.262636 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.262599    2579 scope.go:117] "RemoveContainer" containerID="6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0"
Jul 16 17:57:42.262951 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:57:42.262930    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0\": container with ID starting with 6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0 not found: ID does not exist" containerID="6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0"
Jul 16 17:57:42.263064 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.262955    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0"} err="failed to get container status \"6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0\": rpc error: code = NotFound desc = could not find container \"6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0\": container with ID starting with 6b373fd3018f4a3b171e71b0bdc4f969aca787f1db44a5b0b06e83e0fa7ecec0 not found: ID does not exist"
Jul 16 17:57:42.263064 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.262975    2579 scope.go:117] "RemoveContainer" containerID="047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a"
Jul 16 17:57:42.263205 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:57:42.263187    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a\": container with ID starting with 047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a not found: ID does not exist" containerID="047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a"
Jul 16 17:57:42.263290 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.263210    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a"} err="failed to get container status \"047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a\": rpc error: code = NotFound desc = could not find container \"047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a\": container with ID starting with 047e8f11ea0174c5379b1b40d2f991b11ae0a7b7ffb4598826e12d80d131a80a not found: ID does not exist"
Jul 16 17:57:42.263290 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.263223    2579 scope.go:117] "RemoveContainer" containerID="fd14e4795240bd9890631a916eaeeed8b2e03958c97ee1bd6c4d1ab5c7ce82af"
Jul 16 17:57:42.273572 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.273545    2579 scope.go:117] "RemoveContainer" containerID="9e0687b62e81c33571ffb2de60ab7e2d305454128611141bebba8440321c685b"
Jul 16 17:57:42.274066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.274043    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"]
Jul 16 17:57:42.280005 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.279977    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-869d748779-67hnj"]
Jul 16 17:57:42.296728 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.296699    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"]
Jul 16 17:57:42.304968 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:42.304939    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-5c94d555f-2jzf7"]
Jul 16 17:57:43.104747 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:43.104693    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" path="/var/lib/kubelet/pods/7c54b169-1b7a-4f69-b014-31dab894a2de/volumes"
Jul 16 17:57:43.105479 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:43.105457    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29f8434-84fe-426d-8529-0508870a20eb" path="/var/lib/kubelet/pods/f29f8434-84fe-426d-8529-0508870a20eb/volumes"
Jul 16 17:57:43.237522 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:43.237475    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:57:53.237359 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:57:53.237314    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:58:03.237048 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:58:03.237000    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:58:13.237287 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:58:13.237220    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:58:23.237187 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:58:23.237147    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:58:33.237574 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:58:33.237473    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": dial tcp 10.133.0.52:8000: connect: connection refused"
Jul 16 17:58:43.247373 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:58:43.247337    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:58:43.255083 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:58:43.255052    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:59:08.487948 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:08.487911    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"]
Jul 16 17:59:23.511217 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:23.511149    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main" containerID="cri-o://6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217" gracePeriod=45
Jul 16 17:59:24.173850 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.173827    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:59:24.230996 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.230911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tmp-dir\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.230996 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.230948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-dshm\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.230996 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.230981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-cabundle-cert\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.231244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231002    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-home\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.231244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231018    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pw64h\" (UniqueName: \"kubernetes.io/projected/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kube-api-access-pw64h\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.231244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231066    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kserve-provision-location\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.231244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231085    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-model-cache\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.231244 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231136    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tls-certs\") pod \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\" (UID: \"f582090b-fdb6-4c17-a8bd-e0c3bdbed948\") "
Jul 16 17:59:24.231540 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231435    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:59:24.231600 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231511    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-home" (OuterVolumeSpecName: "home") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:24.231600 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.231552    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-model-cache" (OuterVolumeSpecName: "model-cache") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:24.233289 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.233247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-dshm" (OuterVolumeSpecName: "dshm") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:24.233864 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.233846    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:59:24.233951 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.233864    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kube-api-access-pw64h" (OuterVolumeSpecName: "kube-api-access-pw64h") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "kube-api-access-pw64h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:59:24.248079 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.248047    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:24.289864 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.289825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f582090b-fdb6-4c17-a8bd-e0c3bdbed948" (UID: "f582090b-fdb6-4c17-a8bd-e0c3bdbed948"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:59:24.332312 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332261    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332312 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332307    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332312 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332316    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-cabundle-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332326    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332335    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw64h\" (UniqueName: \"kubernetes.io/projected/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kube-api-access-pw64h\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332344    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332352    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.332541 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.332361    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f582090b-fdb6-4c17-a8bd-e0c3bdbed948-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 17:59:24.858066 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.858028    2579 generic.go:356] "Generic (PLEG): container finished" podID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerID="6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217" exitCode=0
Jul 16 17:59:24.858613 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.858103    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"
Jul 16 17:59:24.858613 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.858120    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" event={"ID":"f582090b-fdb6-4c17-a8bd-e0c3bdbed948","Type":"ContainerDied","Data":"6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217"}
Jul 16 17:59:24.858613 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.858177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn" event={"ID":"f582090b-fdb6-4c17-a8bd-e0c3bdbed948","Type":"ContainerDied","Data":"47dd726945269961611964c1ba6fbe254d9f8c466a7c4d5f5837fcf9b344ce45"}
Jul 16 17:59:24.858613 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.858201    2579 scope.go:117] "RemoveContainer" containerID="6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217"
Jul 16 17:59:24.867843 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.867825    2579 scope.go:117] "RemoveContainer" containerID="415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43"
Jul 16 17:59:24.876465 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.876448    2579 scope.go:117] "RemoveContainer" containerID="6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217"
Jul 16 17:59:24.876720 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:59:24.876701    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217\": container with ID starting with 6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217 not found: ID does not exist" containerID="6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217"
Jul 16 17:59:24.876768 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.876729    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217"} err="failed to get container status \"6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217\": rpc error: code = NotFound desc = could not find container \"6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217\": container with ID starting with 6a40c7ee7612f64453f5418c3e935d1135008e64fbada7eed17f8ddf0123e217 not found: ID does not exist"
Jul 16 17:59:24.876768 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.876746    2579 scope.go:117] "RemoveContainer" containerID="415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43"
Jul 16 17:59:24.876954 ip-10-0-132-31 kubenswrapper[2579]: E0716 17:59:24.876935    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43\": container with ID starting with 415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43 not found: ID does not exist" containerID="415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43"
Jul 16 17:59:24.876994 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.876960    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43"} err="failed to get container status \"415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43\": rpc error: code = NotFound desc = could not find container \"415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43\": container with ID starting with 415a57a42942e342fdb7a04fe539eeb34d28dec2282f2ea4c2e776923a8afc43 not found: ID does not exist"
Jul 16 17:59:24.885904 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.885877    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"]
Jul 16 17:59:24.895011 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:24.894983    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6448b6f56-k8nmn"]
Jul 16 17:59:25.103076 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:25.103033    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" path="/var/lib/kubelet/pods/f582090b-fdb6-4c17-a8bd-e0c3bdbed948/volumes"
Jul 16 17:59:48.201347 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:48.201312    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:59:48.203459 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:48.203439    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 17:59:53.111745 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.111713    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"]
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112062    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112073    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112081    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="storage-initializer"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112088    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="storage-initializer"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112102    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="storage-initializer"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112108    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="storage-initializer"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112114    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="storage-initializer"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112119    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="storage-initializer"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112134    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="main"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112140    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="main"
Jul 16 17:59:53.112144 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112148    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main"
Jul 16 17:59:53.112542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112153    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main"
Jul 16 17:59:53.112542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112209    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c54b169-1b7a-4f69-b014-31dab894a2de" containerName="main"
Jul 16 17:59:53.112542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112218    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29f8434-84fe-426d-8529-0508870a20eb" containerName="main"
Jul 16 17:59:53.112542 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.112224    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f582090b-fdb6-4c17-a8bd-e0c3bdbed948" containerName="main"
Jul 16 17:59:53.116444 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.116426    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.119504 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.119476    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 17:59:53.119652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.119476    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:59:53.119652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.119481    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:59:53.119652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.119569    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 16 17:59:53.119652 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.119480    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:59:53.119869 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.119856    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:59:53.127418 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.127392    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"]
Jul 16 17:59:53.187231 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187196    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187231 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187255    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a96f13-40a6-4ef3-8505-abe92693d0f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cl84\" (UniqueName: \"kubernetes.io/projected/93a96f13-40a6-4ef3-8505-abe92693d0f6-kube-api-access-4cl84\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187480 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187441    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a96f13-40a6-4ef3-8505-abe92693d0f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.187648 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.187502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288668 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288668 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a96f13-40a6-4ef3-8505-abe92693d0f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cl84\" (UniqueName: \"kubernetes.io/projected/93a96f13-40a6-4ef3-8505-abe92693d0f6-kube-api-access-4cl84\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288764    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a96f13-40a6-4ef3-8505-abe92693d0f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.288924 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.288875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.289189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.289069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.289189 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.289104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.289436 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.289415    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.289535 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.289521    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.289634 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.289609    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a96f13-40a6-4ef3-8505-abe92693d0f6-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.291204 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.291189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.291284 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.291208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a96f13-40a6-4ef3-8505-abe92693d0f6-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.299072 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.299051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cl84\" (UniqueName: \"kubernetes.io/projected/93a96f13-40a6-4ef3-8505-abe92693d0f6-kube-api-access-4cl84\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.428454 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.428361    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 17:59:53.576366 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:53.576332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"]
Jul 16 17:59:53.577820 ip-10-0-132-31 kubenswrapper[2579]: W0716 17:59:53.577790    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a96f13_40a6_4ef3_8505_abe92693d0f6.slice/crio-7ea3890ed9522232117bf9ed2d43f0589596b944b10a5f516891eb312232497d WatchSource:0}: Error finding container 7ea3890ed9522232117bf9ed2d43f0589596b944b10a5f516891eb312232497d: Status 404 returned error can't find the container with id 7ea3890ed9522232117bf9ed2d43f0589596b944b10a5f516891eb312232497d
Jul 16 17:59:54.021818 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:54.021778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" event={"ID":"93a96f13-40a6-4ef3-8505-abe92693d0f6","Type":"ContainerStarted","Data":"facf43350df0ad8db5799e90e34db67877315b2d84f3b4e5bee531f450ccdafa"}
Jul 16 17:59:54.021996 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:54.021842    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" event={"ID":"93a96f13-40a6-4ef3-8505-abe92693d0f6","Type":"ContainerStarted","Data":"7ea3890ed9522232117bf9ed2d43f0589596b944b10a5f516891eb312232497d"}
Jul 16 17:59:58.054024 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:58.053993    2579 generic.go:356] "Generic (PLEG): container finished" podID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerID="facf43350df0ad8db5799e90e34db67877315b2d84f3b4e5bee531f450ccdafa" exitCode=0
Jul 16 17:59:58.054536 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:58.054293    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" event={"ID":"93a96f13-40a6-4ef3-8505-abe92693d0f6","Type":"ContainerDied","Data":"facf43350df0ad8db5799e90e34db67877315b2d84f3b4e5bee531f450ccdafa"}
Jul 16 17:59:59.065848 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:59.065818    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" event={"ID":"93a96f13-40a6-4ef3-8505-abe92693d0f6","Type":"ContainerStarted","Data":"d4baa02c301c1edf1e3d7b08669686b49eca31d4fe2ad14e703014235b382888"}
Jul 16 17:59:59.097930 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:59.097875    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podStartSLOduration=2.486904607 podStartE2EDuration="6.09785278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.610948173s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:59:59.093863075 +0000 UTC m=+1520.477333008" watchObservedRunningTime="2026-07-16 17:59:59.09785278 +0000 UTC m=+1520.481322717"
Jul 16 17:59:59.429154 ip-10-0-132-31 kubenswrapper[2579]: I0716 17:59:59.429051    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 18:00:03.429045 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:03.428948    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 18:00:03.430696 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:03.430662    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:00:13.429818 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:13.429771    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:00:23.429764 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:23.429713    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:00:33.429001 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:33.428959    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:00:43.429105 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:43.429047    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:00:53.429622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:00:53.429575    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:01:03.428769 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:03.428728    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:01:13.429671 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:13.429620    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:01:23.439802 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:23.439766    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 18:01:23.448086 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:23.448061    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 18:01:25.016147 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:25.016114    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"]
Jul 16 18:01:40.040898 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.040785    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" containerID="cri-o://d4baa02c301c1edf1e3d7b08669686b49eca31d4fe2ad14e703014235b382888" gracePeriod=45
Jul 16 18:01:40.441520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.441413    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 16 18:01:40.640725 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.640692    2579 generic.go:356] "Generic (PLEG): container finished" podID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerID="d4baa02c301c1edf1e3d7b08669686b49eca31d4fe2ad14e703014235b382888" exitCode=0
Jul 16 18:01:40.640907 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.640768    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" event={"ID":"93a96f13-40a6-4ef3-8505-abe92693d0f6","Type":"ContainerDied","Data":"d4baa02c301c1edf1e3d7b08669686b49eca31d4fe2ad14e703014235b382888"}
Jul 16 18:01:40.694582 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.694502    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 18:01:40.890801 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890766    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a96f13-40a6-4ef3-8505-abe92693d0f6-cabundle-cert\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.890801 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890811    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-tmp-dir\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-dshm\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890900    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-home\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890927    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cl84\" (UniqueName: \"kubernetes.io/projected/93a96f13-40a6-4ef3-8505-abe92693d0f6-kube-api-access-4cl84\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890952    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-kserve-provision-location\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890971    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a96f13-40a6-4ef3-8505-abe92693d0f6-tls-certs\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.890987    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-model-cache\") pod \"93a96f13-40a6-4ef3-8505-abe92693d0f6\" (UID: \"93a96f13-40a6-4ef3-8505-abe92693d0f6\") "
Jul 16 18:01:40.891478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.891239    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a96f13-40a6-4ef3-8505-abe92693d0f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:01:40.891478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.891312    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.891478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.891466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.891661 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.891494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-home" (OuterVolumeSpecName: "home") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.893151 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.893120    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-dshm" (OuterVolumeSpecName: "dshm") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.893256 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.893164    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a96f13-40a6-4ef3-8505-abe92693d0f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:01:40.893256 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.893189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a96f13-40a6-4ef3-8505-abe92693d0f6-kube-api-access-4cl84" (OuterVolumeSpecName: "kube-api-access-4cl84") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "kube-api-access-4cl84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:01:40.961079 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.960995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "93a96f13-40a6-4ef3-8505-abe92693d0f6" (UID: "93a96f13-40a6-4ef3-8505-abe92693d0f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:01:40.991898 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991859    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/93a96f13-40a6-4ef3-8505-abe92693d0f6-cabundle-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.991898 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991894    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.991898 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991905    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.992132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991915    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.992132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991926    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cl84\" (UniqueName: \"kubernetes.io/projected/93a96f13-40a6-4ef3-8505-abe92693d0f6-kube-api-access-4cl84\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.992132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991939    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.992132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991951    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a96f13-40a6-4ef3-8505-abe92693d0f6-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:40.992132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:40.991963    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a96f13-40a6-4ef3-8505-abe92693d0f6-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:01:41.647557 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:41.647514    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d" event={"ID":"93a96f13-40a6-4ef3-8505-abe92693d0f6","Type":"ContainerDied","Data":"7ea3890ed9522232117bf9ed2d43f0589596b944b10a5f516891eb312232497d"}
Jul 16 18:01:41.647980 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:41.647578    2579 scope.go:117] "RemoveContainer" containerID="d4baa02c301c1edf1e3d7b08669686b49eca31d4fe2ad14e703014235b382888"
Jul 16 18:01:41.647980 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:41.647531    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"
Jul 16 18:01:41.656640 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:41.656616    2579 scope.go:117] "RemoveContainer" containerID="facf43350df0ad8db5799e90e34db67877315b2d84f3b4e5bee531f450ccdafa"
Jul 16 18:01:41.672409 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:41.672385    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"]
Jul 16 18:01:41.677924 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:41.677898    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-c8bdfd4c5-b7t9d"]
Jul 16 18:01:43.103577 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:43.103539    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" path="/var/lib/kubelet/pods/93a96f13-40a6-4ef3-8505-abe92693d0f6/volumes"
Jul 16 18:01:48.416933 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.416892    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"]
Jul 16 18:01:48.417390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.417255    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="storage-initializer"
Jul 16 18:01:48.417390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.417279    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="storage-initializer"
Jul 16 18:01:48.417390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.417290    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main"
Jul 16 18:01:48.417390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.417296    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main"
Jul 16 18:01:48.417390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.417358    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="93a96f13-40a6-4ef3-8505-abe92693d0f6" containerName="main"
Jul 16 18:01:48.420239 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.420222    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:48.423511 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.423490    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:01:48.423511 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.423495    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:01:48.423680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.423523    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:01:48.428470 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.428240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"]
Jul 16 18:01:48.549285 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.549228    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssg79\" (UniqueName: \"kubernetes.io/projected/9a9a2926-049a-43b6-b6bc-41fcfe89713a-kube-api-access-ssg79\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:48.650660 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.650626    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssg79\" (UniqueName: \"kubernetes.io/projected/9a9a2926-049a-43b6-b6bc-41fcfe89713a-kube-api-access-ssg79\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:48.661273 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:48.661244    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssg79\" (UniqueName: \"kubernetes.io/projected/9a9a2926-049a-43b6-b6bc-41fcfe89713a-kube-api-access-ssg79\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.475835 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.475791    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.491790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.491751    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") device path: \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.576344 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.576301    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.576523 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.576431    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") DevicePath \"\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.580285 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.580244    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.965683 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.965647    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b45f7228e14240b65ad12bdfed958115c57791e355291dc13fb10bdd9eb08268/globalmount\"" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:51.980283 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:51.980247    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"e2e-pvc-model-download-v8gm9\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") " pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:52.030818 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:52.030779    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:52.159807 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:52.159781    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"]
Jul 16 18:01:52.161502 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:01:52.161465    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a9a2926_049a_43b6_b6bc_41fcfe89713a.slice/crio-3b1faf31da22fc7ff6ab7fa8e4039432628965668ab06e28b1e50b83b40c02da WatchSource:0}: Error finding container 3b1faf31da22fc7ff6ab7fa8e4039432628965668ab06e28b1e50b83b40c02da: Status 404 returned error can't find the container with id 3b1faf31da22fc7ff6ab7fa8e4039432628965668ab06e28b1e50b83b40c02da
Jul 16 18:01:52.703999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:52.703951    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9" event={"ID":"9a9a2926-049a-43b6-b6bc-41fcfe89713a","Type":"ContainerStarted","Data":"aff56ceba3ca3bf4e4ad703f31f8e45c1a69f15068b222522778f7ccea727677"}
Jul 16 18:01:52.703999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:52.704004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9" event={"ID":"9a9a2926-049a-43b6-b6bc-41fcfe89713a","Type":"ContainerStarted","Data":"3b1faf31da22fc7ff6ab7fa8e4039432628965668ab06e28b1e50b83b40c02da"}
Jul 16 18:01:52.720974 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:52.720927    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9" podStartSLOduration=7.720911307 podStartE2EDuration="7.720911307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:01:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:52.719893665 +0000 UTC m=+1634.103363599" watchObservedRunningTime="2026-07-16 18:01:52.720911307 +0000 UTC m=+1634.104381287"
Jul 16 18:01:58.738474 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:58.738434    2579 generic.go:356] "Generic (PLEG): container finished" podID="9a9a2926-049a-43b6-b6bc-41fcfe89713a" containerID="aff56ceba3ca3bf4e4ad703f31f8e45c1a69f15068b222522778f7ccea727677" exitCode=0
Jul 16 18:01:58.738890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:58.738501    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9" event={"ID":"9a9a2926-049a-43b6-b6bc-41fcfe89713a","Type":"ContainerDied","Data":"aff56ceba3ca3bf4e4ad703f31f8e45c1a69f15068b222522778f7ccea727677"}
Jul 16 18:01:59.886053 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:59.886028    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:01:59.955896 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:59.955865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-storage\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") "
Jul 16 18:01:59.956064 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:59.955913    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssg79\" (UniqueName: \"kubernetes.io/projected/9a9a2926-049a-43b6-b6bc-41fcfe89713a-kube-api-access-ssg79\") pod \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\" (UID: \"9a9a2926-049a-43b6-b6bc-41fcfe89713a\") "
Jul 16 18:01:59.958022 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:59.957997    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a9a2926-049a-43b6-b6bc-41fcfe89713a-kube-api-access-ssg79" (OuterVolumeSpecName: "kube-api-access-ssg79") pod "9a9a2926-049a-43b6-b6bc-41fcfe89713a" (UID: "9a9a2926-049a-43b6-b6bc-41fcfe89713a"). InnerVolumeSpecName "kube-api-access-ssg79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:01:59.959622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:01:59.959600    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f" (OuterVolumeSpecName: "model-storage") pod "9a9a2926-049a-43b6-b6bc-41fcfe89713a" (UID: "9a9a2926-049a-43b6-b6bc-41fcfe89713a"). InnerVolumeSpecName "pvc-816e24b0-2442-4458-a346-0c49c6eaace7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 18:02:00.057416 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.057389    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" "
Jul 16 18:02:00.057416 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.057420    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssg79\" (UniqueName: \"kubernetes.io/projected/9a9a2926-049a-43b6-b6bc-41fcfe89713a-kube-api-access-ssg79\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:02:00.176843 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.176818    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-816e24b0-2442-4458-a346-0c49c6eaace7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f") on node "ip-10-0-132-31.ec2.internal"
Jul 16 18:02:00.259295 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.259243    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\""
Jul 16 18:02:00.749885 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.749846    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9" event={"ID":"9a9a2926-049a-43b6-b6bc-41fcfe89713a","Type":"ContainerDied","Data":"3b1faf31da22fc7ff6ab7fa8e4039432628965668ab06e28b1e50b83b40c02da"}
Jul 16 18:02:00.749885 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.749867    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/e2e-pvc-model-download-v8gm9"
Jul 16 18:02:00.749885 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:00.749889    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b1faf31da22fc7ff6ab7fa8e4039432628965668ab06e28b1e50b83b40c02da"
Jul 16 18:02:13.992558 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.992516    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"]
Jul 16 18:02:13.992979 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.992880    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a9a2926-049a-43b6-b6bc-41fcfe89713a" containerName="storage-initializer"
Jul 16 18:02:13.992979 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.992892    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a9a2926-049a-43b6-b6bc-41fcfe89713a" containerName="storage-initializer"
Jul 16 18:02:13.992979 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.992961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a9a2926-049a-43b6-b6bc-41fcfe89713a" containerName="storage-initializer"
Jul 16 18:02:13.994896 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.994879    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:13.998045 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.998020    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:02:13.998154 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.998046    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:02:13.998154 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.998054    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:02:13.998154 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:13.998056    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 16 18:02:14.009160 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.009135    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"]
Jul 16 18:02:14.079858 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.079825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.080026 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.079894    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.080026 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.079927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.080026 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.079947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.080026 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.079990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zskf9\" (UniqueName: \"kubernetes.io/projected/8d40334b-f7ff-48b9-b4b1-11a69b76209b-kube-api-access-zskf9\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.080026 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.080023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.180561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180519    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.180757 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180577    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.180757 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.180757 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180647    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zskf9\" (UniqueName: \"kubernetes.io/projected/8d40334b-f7ff-48b9-b4b1-11a69b76209b-kube-api-access-zskf9\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.180958 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.180958 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180942    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.181077 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.180973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.181077 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.181017    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.181646 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.181615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.184235 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.184205    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.184375 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.184345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:14.190922 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:14.190894    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskf9\" (UniqueName: \"kubernetes.io/projected/8d40334b-f7ff-48b9-b4b1-11a69b76209b-kube-api-access-zskf9\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.661192 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.661153    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.669167 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.669131    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.762325 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.762260    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.762522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.762369    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.769400 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.769377    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.893736 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.893702    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b45f7228e14240b65ad12bdfed958115c57791e355291dc13fb10bdd9eb08268/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:21.924244 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:21.924163    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:22.104982 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:22.104942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:22.241084 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:22.241053    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"]
Jul 16 18:02:22.243803 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:02:22.243775    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d40334b_f7ff_48b9_b4b1_11a69b76209b.slice/crio-4a36aa1e6417f2a3f6000c4af93a73883e8479b12ca86e59b2607b9a3275bb65 WatchSource:0}: Error finding container 4a36aa1e6417f2a3f6000c4af93a73883e8479b12ca86e59b2607b9a3275bb65: Status 404 returned error can't find the container with id 4a36aa1e6417f2a3f6000c4af93a73883e8479b12ca86e59b2607b9a3275bb65
Jul 16 18:02:22.878824 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:22.878782    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" event={"ID":"8d40334b-f7ff-48b9-b4b1-11a69b76209b","Type":"ContainerStarted","Data":"4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136"}
Jul 16 18:02:22.879199 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:22.878849    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" event={"ID":"8d40334b-f7ff-48b9-b4b1-11a69b76209b","Type":"ContainerStarted","Data":"4a36aa1e6417f2a3f6000c4af93a73883e8479b12ca86e59b2607b9a3275bb65"}
Jul 16 18:02:22.905922 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:22.905860    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podStartSLOduration=9.905838687 podStartE2EDuration="9.905838687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:02:22.900623237 +0000 UTC m=+1664.284093164" watchObservedRunningTime="2026-07-16 18:02:22.905838687 +0000 UTC m=+1664.289308622"
Jul 16 18:02:23.105071 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:23.105041    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:32.105767 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:32.105729    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:02:32.107614 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:32.107581    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:02:42.106222 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:42.106171    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:02:52.106182 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:02:52.106136    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:03:02.106226 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:02.106132    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:03:12.106285 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:12.106223    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:03:22.105981 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:22.105935    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:03:30.712607 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.712569    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"]
Jul 16 18:03:30.716216 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.716197    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.719447 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.719414    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:03:30.719566 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.719454    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 16 18:03:30.727725 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.727700    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"]
Jul 16 18:03:30.808731 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.808684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.808731 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.808734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2km7d\" (UniqueName: \"kubernetes.io/projected/0f0ea46c-1911-4bd5-9b81-2998e5261624-kube-api-access-2km7d\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.809000 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.808816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.809000 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.808847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0ea46c-1911-4bd5-9b81-2998e5261624-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.809000 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.808904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.809000 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.808977    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.809204 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.809000    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.909986    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910050    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910119    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2km7d\" (UniqueName: \"kubernetes.io/projected/0f0ea46c-1911-4bd5-9b81-2998e5261624-kube-api-access-2km7d\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0ea46c-1911-4bd5-9b81-2998e5261624-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910609 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910688 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910610    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910688 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.910761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.910700    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.912511 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.912489    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.912682 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.912665    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0ea46c-1911-4bd5-9b81-2998e5261624-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:30.920099 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:30.920074    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2km7d\" (UniqueName: \"kubernetes.io/projected/0f0ea46c-1911-4bd5-9b81-2998e5261624-kube-api-access-2km7d\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:31.028421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:31.028326    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:31.179811 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:31.179782    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"]
Jul 16 18:03:31.182679 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:03:31.182651    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0ea46c_1911_4bd5_9b81_2998e5261624.slice/crio-23ce6ffa01713390265582a27ab0e88622a25e306956c16ae559e06372387b93 WatchSource:0}: Error finding container 23ce6ffa01713390265582a27ab0e88622a25e306956c16ae559e06372387b93: Status 404 returned error can't find the container with id 23ce6ffa01713390265582a27ab0e88622a25e306956c16ae559e06372387b93
Jul 16 18:03:31.270771 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:31.270499    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" event={"ID":"0f0ea46c-1911-4bd5-9b81-2998e5261624","Type":"ContainerStarted","Data":"c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3"}
Jul 16 18:03:31.270771 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:31.270568    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" event={"ID":"0f0ea46c-1911-4bd5-9b81-2998e5261624","Type":"ContainerStarted","Data":"23ce6ffa01713390265582a27ab0e88622a25e306956c16ae559e06372387b93"}
Jul 16 18:03:32.106348 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:32.106294    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:03:34.294595 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:34.294564    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerID="c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3" exitCode=0
Jul 16 18:03:34.294949 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:34.294636    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" event={"ID":"0f0ea46c-1911-4bd5-9b81-2998e5261624","Type":"ContainerDied","Data":"c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3"}
Jul 16 18:03:35.304035 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:35.303999    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" event={"ID":"0f0ea46c-1911-4bd5-9b81-2998e5261624","Type":"ContainerStarted","Data":"8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306"}
Jul 16 18:03:35.331417 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:35.331347    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podStartSLOduration=2.318118942 podStartE2EDuration="5.331324544s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.013205602s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:03:35.327912034 +0000 UTC m=+1736.711381969" watchObservedRunningTime="2026-07-16 18:03:35.331324544 +0000 UTC m=+1736.714794479"
Jul 16 18:03:36.028873 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:36.028839    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:41.028692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:41.028641    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:03:41.030556 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:41.030516    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:03:42.106441 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:42.106382    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:03:51.028851 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:51.028798    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:03:52.106010 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:03:52.105969    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:04:01.029129 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:01.029078    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:04:02.106059 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:02.106005    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:04:11.029698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:11.029653    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:04:12.105991 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:12.105934    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 16 18:04:21.029292 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:21.029223    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:04:22.115985 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:22.115945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:04:22.123816 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:22.123793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:04:27.335766 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:27.335731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"]
Jul 16 18:04:31.028846 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:31.028708    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:04:41.029770 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:41.029726    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:04:42.359936 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:42.359888    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main" containerID="cri-o://4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136" gracePeriod=45
Jul 16 18:04:43.115447 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.114592    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:04:43.198796 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.198702    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-dshm\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.198796 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.198746    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tls-certs\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.199034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.198845    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.199034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.198873    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-home\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.199034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.198981    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tmp-dir\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.199034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.199022    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-model-cache\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.199248 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.199077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zskf9\" (UniqueName: \"kubernetes.io/projected/8d40334b-f7ff-48b9-b4b1-11a69b76209b-kube-api-access-zskf9\") pod \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\" (UID: \"8d40334b-f7ff-48b9-b4b1-11a69b76209b\") "
Jul 16 18:04:43.199382 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.199352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-model-cache" (OuterVolumeSpecName: "model-cache") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:04:43.199589 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.199562    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-home" (OuterVolumeSpecName: "home") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:04:43.201599 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.201559    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:04:43.201822 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.201794    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-dshm" (OuterVolumeSpecName: "dshm") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:04:43.201939 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.201814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d40334b-f7ff-48b9-b4b1-11a69b76209b-kube-api-access-zskf9" (OuterVolumeSpecName: "kube-api-access-zskf9") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "kube-api-access-zskf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:04:43.202128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.202109    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f" (OuterVolumeSpecName: "kserve-pvc-source") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "pvc-816e24b0-2442-4458-a346-0c49c6eaace7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 18:04:43.210531 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.210499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8d40334b-f7ff-48b9-b4b1-11a69b76209b" (UID: "8d40334b-f7ff-48b9-b4b1-11a69b76209b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:04:43.300722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300683    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.300722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300720    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zskf9\" (UniqueName: \"kubernetes.io/projected/8d40334b-f7ff-48b9-b4b1-11a69b76209b-kube-api-access-zskf9\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.300972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300734    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.300972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300747    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.300972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300786    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" "
Jul 16 18:04:43.300972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300797    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.300972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.300809    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d40334b-f7ff-48b9-b4b1-11a69b76209b-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:04:43.328141 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.328117    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-816e24b0-2442-4458-a346-0c49c6eaace7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f") on node "ip-10-0-132-31.ec2.internal"
Jul 16 18:04:43.401432 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.401397    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\""
Jul 16 18:04:43.754399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.754364    2579 generic.go:356] "Generic (PLEG): container finished" podID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerID="4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136" exitCode=0
Jul 16 18:04:43.754592 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.754442    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"
Jul 16 18:04:43.754592 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.754441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" event={"ID":"8d40334b-f7ff-48b9-b4b1-11a69b76209b","Type":"ContainerDied","Data":"4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136"}
Jul 16 18:04:43.754592 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.754588    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff" event={"ID":"8d40334b-f7ff-48b9-b4b1-11a69b76209b","Type":"ContainerDied","Data":"4a36aa1e6417f2a3f6000c4af93a73883e8479b12ca86e59b2607b9a3275bb65"}
Jul 16 18:04:43.754723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.754611    2579 scope.go:117] "RemoveContainer" containerID="4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136"
Jul 16 18:04:43.763880 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.763862    2579 scope.go:117] "RemoveContainer" containerID="4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136"
Jul 16 18:04:43.764210 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:04:43.764183    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136\": container with ID starting with 4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136 not found: ID does not exist" containerID="4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136"
Jul 16 18:04:43.764336 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.764222    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136"} err="failed to get container status \"4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136\": rpc error: code = NotFound desc = could not find container \"4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136\": container with ID starting with 4a58a9ec55549897ff7c0f4adc2559f96c853bc8ef7d9f81b8a60644eb2e7136 not found: ID does not exist"
Jul 16 18:04:43.783829 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.783802    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"]
Jul 16 18:04:43.794528 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:43.794502    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9dcbpff"]
Jul 16 18:04:45.105066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:45.105025    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" path="/var/lib/kubelet/pods/8d40334b-f7ff-48b9-b4b1-11a69b76209b/volumes"
Jul 16 18:04:48.239585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:48.239549    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:04:48.243143 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:48.243121    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:04:51.029436 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:51.029386    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:04:57.147395 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.147358    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"]
Jul 16 18:04:57.147965 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.147923    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main"
Jul 16 18:04:57.147965 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.147945    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main"
Jul 16 18:04:57.148078 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.148069    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d40334b-f7ff-48b9-b4b1-11a69b76209b" containerName="main"
Jul 16 18:04:57.152211 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.152186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.155254 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.155224    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-5cw8j\""
Jul 16 18:04:57.155474 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.155445    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 16 18:04:57.166357 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.166311    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"]
Jul 16 18:04:57.174704 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.174675    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"]
Jul 16 18:04:57.177321 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.177296    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.192938 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.192910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"]
Jul 16 18:04:57.227260 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.227467 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227309    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.227467 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227336    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.227467 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227362    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.227577 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227471    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.227577 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9gnc\" (UniqueName: \"kubernetes.io/projected/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-kube-api-access-s9gnc\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.227645 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b15921-a3fc-4f4b-9366-a3d88805ed03-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.227679 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.227714 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thjn\" (UniqueName: \"kubernetes.io/projected/51b15921-a3fc-4f4b-9366-a3d88805ed03-kube-api-access-8thjn\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.227751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.227788 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227761    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.227788 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.227782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329261 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b15921-a3fc-4f4b-9366-a3d88805ed03-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.329466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.329466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329316    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thjn\" (UniqueName: \"kubernetes.io/projected/51b15921-a3fc-4f4b-9366-a3d88805ed03-kube-api-access-8thjn\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.329466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329369    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.329809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.329809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329518    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.329809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9gnc\" (UniqueName: \"kubernetes.io/projected/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-kube-api-access-s9gnc\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.329809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329761    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.330128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.329826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.330128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.330029    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.330128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.330057    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.330128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.330111    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.330339 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.330309    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.332190 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.332167    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.332328 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.332230    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.332328 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.332229    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:04:57.332561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.332542    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b15921-a3fc-4f4b-9366-a3d88805ed03-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.339644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.339620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thjn\" (UniqueName: \"kubernetes.io/projected/51b15921-a3fc-4f4b-9366-a3d88805ed03-kube-api-access-8thjn\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:04:57.340133 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:04:57.340112    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9gnc\" (UniqueName: \"kubernetes.io/projected/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-kube-api-access-s9gnc\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:01.029542 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:01.029491    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:05:05.309978 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.309938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.317606 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.317576    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.411514 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.411471    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.411699 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.411635    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.415464 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.415443    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.509147 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.509111    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b45f7228e14240b65ad12bdfed958115c57791e355291dc13fb10bdd9eb08268/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.527890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.527860    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.568094 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.568001    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:05.614072 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.614028    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:05.614298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.614182    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:05.619130 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.618906    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:05.641370 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.641249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:05.717626 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.717595    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"]
Jul 16 18:05:05.718627 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:05:05.718600    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b15921_a3fc_4f4b_9366_a3d88805ed03.slice/crio-61fd47b4eb535e904fe5000caed0ce00c6bc5ee1698a7ea707252f8cec3643ca WatchSource:0}: Error finding container 61fd47b4eb535e904fe5000caed0ce00c6bc5ee1698a7ea707252f8cec3643ca: Status 404 returned error can't find the container with id 61fd47b4eb535e904fe5000caed0ce00c6bc5ee1698a7ea707252f8cec3643ca
Jul 16 18:05:05.720601 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.720585    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 18:05:05.889441 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.889345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:05.892319 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:05.892294    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" event={"ID":"51b15921-a3fc-4f4b-9366-a3d88805ed03","Type":"ContainerStarted","Data":"61fd47b4eb535e904fe5000caed0ce00c6bc5ee1698a7ea707252f8cec3643ca"}
Jul 16 18:05:06.037557 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:06.037532    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"]
Jul 16 18:05:06.039326 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:05:06.039300    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45871f3c_410a_4e86_8b6b_6a5f29f7a9e1.slice/crio-8ada49568fb0d8bb7aadb4cf8f3723ffd28f6fa3bb0a9030ba898a907b8f174e WatchSource:0}: Error finding container 8ada49568fb0d8bb7aadb4cf8f3723ffd28f6fa3bb0a9030ba898a907b8f174e: Status 404 returned error can't find the container with id 8ada49568fb0d8bb7aadb4cf8f3723ffd28f6fa3bb0a9030ba898a907b8f174e
Jul 16 18:05:06.901874 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:06.901831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" event={"ID":"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1","Type":"ContainerStarted","Data":"68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca"}
Jul 16 18:05:06.902438 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:06.901902    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" event={"ID":"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1","Type":"ContainerStarted","Data":"8ada49568fb0d8bb7aadb4cf8f3723ffd28f6fa3bb0a9030ba898a907b8f174e"}
Jul 16 18:05:06.928386 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:06.928322    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podStartSLOduration=9.928299464 podStartE2EDuration="9.928299464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:05:06.924779066 +0000 UTC m=+1828.308249015" watchObservedRunningTime="2026-07-16 18:05:06.928299464 +0000 UTC m=+1828.311769399"
Jul 16 18:05:07.911106 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:07.911002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" event={"ID":"51b15921-a3fc-4f4b-9366-a3d88805ed03","Type":"ContainerStarted","Data":"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"}
Jul 16 18:05:07.911573 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:07.911384    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:08.920665 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:08.920626    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" event={"ID":"51b15921-a3fc-4f4b-9366-a3d88805ed03","Type":"ContainerStarted","Data":"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"}
Jul 16 18:05:08.950475 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:08.950406    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podStartSLOduration=10.12455644 podStartE2EDuration="11.950382195s" totalImagesPullingTime="1.825825755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:05:08.946574444 +0000 UTC m=+1830.330044379" watchObservedRunningTime="2026-07-16 18:05:08.950382195 +0000 UTC m=+1830.333852131"
Jul 16 18:05:11.029794 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:11.029747    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:05:15.569118 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:15.569070    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:15.569639 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:15.569164    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:15.570795 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:15.570760    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:05:15.890591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:15.890497    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:15.890776 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:15.890753    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:05:15.892502 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:15.892469    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:05:21.029812 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:21.029751    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:05:25.568685 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:25.568635    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:05:25.587395 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:25.587362    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:05:25.890790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:25.890682    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:05:31.045449 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:31.045415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:05:31.057501 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:31.057465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:05:35.569541 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:35.569468    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:05:35.891009 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:35.890885    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:05:35.911188 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:35.911153    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"]
Jul 16 18:05:45.568688 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:45.568639    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:05:45.890041 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:45.889926    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:05:50.940966 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:50.940925    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" containerID="cri-o://8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306" gracePeriod=45
Jul 16 18:05:51.046204 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.046156    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 16 18:05:51.704128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.704082    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:05:51.815835 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.815799    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-home\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816023 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.815854    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0ea46c-1911-4bd5-9b81-2998e5261624-tls-certs\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816023 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.815883    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-model-cache\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816023 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.815955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-kserve-provision-location\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816220 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.816096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-dshm\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816428 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.816400    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.816541 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.816492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2km7d\" (UniqueName: \"kubernetes.io/projected/0f0ea46c-1911-4bd5-9b81-2998e5261624-kube-api-access-2km7d\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.816646    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-tmp-dir\") pod \"0f0ea46c-1911-4bd5-9b81-2998e5261624\" (UID: \"0f0ea46c-1911-4bd5-9b81-2998e5261624\") "
Jul 16 18:05:51.816751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.816722    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-home" (OuterVolumeSpecName: "home") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.817194 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.817170    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.817302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.817199    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.818875 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.818839    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f0ea46c-1911-4bd5-9b81-2998e5261624-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:05:51.818875 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.818856    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-dshm" (OuterVolumeSpecName: "dshm") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.820423 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.820395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0ea46c-1911-4bd5-9b81-2998e5261624-kube-api-access-2km7d" (OuterVolumeSpecName: "kube-api-access-2km7d") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "kube-api-access-2km7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:05:51.830680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.830636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.874113 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.874070    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f0ea46c-1911-4bd5-9b81-2998e5261624" (UID: "0f0ea46c-1911-4bd5-9b81-2998e5261624"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:05:51.917946 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.917902    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.917946 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.917938    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.918217 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.917958    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2km7d\" (UniqueName: \"kubernetes.io/projected/0f0ea46c-1911-4bd5-9b81-2998e5261624-kube-api-access-2km7d\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.918217 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.917971    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f0ea46c-1911-4bd5-9b81-2998e5261624-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:51.918217 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:51.917982    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f0ea46c-1911-4bd5-9b81-2998e5261624-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:05:52.221737 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.221644    2579 generic.go:356] "Generic (PLEG): container finished" podID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerID="8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306" exitCode=0
Jul 16 18:05:52.221737 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.221687    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"
Jul 16 18:05:52.221737 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.221721    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" event={"ID":"0f0ea46c-1911-4bd5-9b81-2998e5261624","Type":"ContainerDied","Data":"8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306"}
Jul 16 18:05:52.222243 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.221778    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2" event={"ID":"0f0ea46c-1911-4bd5-9b81-2998e5261624","Type":"ContainerDied","Data":"23ce6ffa01713390265582a27ab0e88622a25e306956c16ae559e06372387b93"}
Jul 16 18:05:52.222243 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.221811    2579 scope.go:117] "RemoveContainer" containerID="8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306"
Jul 16 18:05:52.233572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.233547    2579 scope.go:117] "RemoveContainer" containerID="c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3"
Jul 16 18:05:52.242390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.242367    2579 scope.go:117] "RemoveContainer" containerID="8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306"
Jul 16 18:05:52.242724 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:05:52.242696    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306\": container with ID starting with 8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306 not found: ID does not exist" containerID="8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306"
Jul 16 18:05:52.242780 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.242738    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306"} err="failed to get container status \"8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306\": rpc error: code = NotFound desc = could not find container \"8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306\": container with ID starting with 8e87cf7e2d6f1bc99bda332673f46008db3a08aaecdae70e729752559757f306 not found: ID does not exist"
Jul 16 18:05:52.242780 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.242763    2579 scope.go:117] "RemoveContainer" containerID="c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3"
Jul 16 18:05:52.243047 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:05:52.243024    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3\": container with ID starting with c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3 not found: ID does not exist" containerID="c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3"
Jul 16 18:05:52.243107 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.243058    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3"} err="failed to get container status \"c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3\": rpc error: code = NotFound desc = could not find container \"c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3\": container with ID starting with c05a26db72bd13f04fbda044c291439a74d23735efe98b7ddbaf5419c901ccf3 not found: ID does not exist"
Jul 16 18:05:52.250354 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.250330    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"]
Jul 16 18:05:52.254171 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:52.254147    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-5bccf9c7bf7rct2"]
Jul 16 18:05:53.104692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:53.104652    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" path="/var/lib/kubelet/pods/0f0ea46c-1911-4bd5-9b81-2998e5261624/volumes"
Jul 16 18:05:55.568814 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:55.568736    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:05:55.889841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:05:55.889739    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:06:05.569081 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:05.568972    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:06:05.890064 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:05.889963    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:06:12.085125 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.085092    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"]
Jul 16 18:06:12.085546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.085493    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="storage-initializer"
Jul 16 18:06:12.085546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.085505    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="storage-initializer"
Jul 16 18:06:12.085546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.085525    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main"
Jul 16 18:06:12.085546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.085531    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main"
Jul 16 18:06:12.085696 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.085595    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f0ea46c-1911-4bd5-9b81-2998e5261624" containerName="main"
Jul 16 18:06:12.089058 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.089038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.091867 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.091842    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:06:12.092011 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.091878    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 16 18:06:12.103463 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.103429    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"]
Jul 16 18:06:12.231626 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqzwl\" (UniqueName: \"kubernetes.io/projected/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kube-api-access-sqzwl\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.231816 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-home\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.231816 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tmp-dir\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.231816 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-model-cache\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.231977 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tls-certs\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.231977 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231904    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-dshm\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.231977 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.231920    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.332875 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.332838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tmp-dir\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.332885    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-model-cache\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.332904    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tls-certs\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.332925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-dshm\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.332945    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333329 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.333110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqzwl\" (UniqueName: \"kubernetes.io/projected/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kube-api-access-sqzwl\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333329 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.333190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-home\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333446 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.333364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-model-cache\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333543 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.333519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333588 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.333536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-home\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.333750 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.333723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tmp-dir\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.335368 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.335303    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-dshm\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.336017 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.335994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tls-certs\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.345930 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.345904    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqzwl\" (UniqueName: \"kubernetes.io/projected/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kube-api-access-sqzwl\") pod \"custom-route-timeout-test-kserve-d4b5cbd7-ntdnx\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.400373 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.400330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:12.569942 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:12.569910    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"]
Jul 16 18:06:12.571777 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:06:12.571747    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee5db9b_eef3_4f9b_a818_1d91d2f783a8.slice/crio-65b8543432416eb9674eee7f6699c0e64b2507a7107ed8d7bcca44f42788ab0c WatchSource:0}: Error finding container 65b8543432416eb9674eee7f6699c0e64b2507a7107ed8d7bcca44f42788ab0c: Status 404 returned error can't find the container with id 65b8543432416eb9674eee7f6699c0e64b2507a7107ed8d7bcca44f42788ab0c
Jul 16 18:06:13.365887 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:13.365852    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" event={"ID":"dee5db9b-eef3-4f9b-a818-1d91d2f783a8","Type":"ContainerStarted","Data":"5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba"}
Jul 16 18:06:13.366391 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:13.365910    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" event={"ID":"dee5db9b-eef3-4f9b-a818-1d91d2f783a8","Type":"ContainerStarted","Data":"65b8543432416eb9674eee7f6699c0e64b2507a7107ed8d7bcca44f42788ab0c"}
Jul 16 18:06:15.569107 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:15.569052    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:06:15.890156 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:15.890050    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:06:16.389100 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:16.389060    2579 generic.go:356] "Generic (PLEG): container finished" podID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerID="5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba" exitCode=0
Jul 16 18:06:16.389286 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:16.389124    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" event={"ID":"dee5db9b-eef3-4f9b-a818-1d91d2f783a8","Type":"ContainerDied","Data":"5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba"}
Jul 16 18:06:17.399759 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:17.399713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" event={"ID":"dee5db9b-eef3-4f9b-a818-1d91d2f783a8","Type":"ContainerStarted","Data":"cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa"}
Jul 16 18:06:17.400545 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:17.400525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:17.427770 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:17.427709    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podStartSLOduration=1.7295389490000002 podStartE2EDuration="5.427689614s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.698150665s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:06:17.424227849 +0000 UTC m=+1898.807697794" watchObservedRunningTime="2026-07-16 18:06:17.427689614 +0000 UTC m=+1898.811159549"
Jul 16 18:06:22.401140 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:22.401100    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:06:22.403133 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:22.403099    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:06:25.569071 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:25.569016    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:06:25.890316 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:25.890207    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:06:32.401382 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:32.401336    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:06:35.569428 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:35.569370    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:06:35.889895 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:35.889783    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:06:42.400892 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:42.400839    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:06:45.568587 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:45.568536    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:06:45.890866 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:45.890763    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:06:52.401289 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:52.401220    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:06:55.568899 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:55.568846    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:06:55.890405 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:06:55.890305    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:07:02.401303 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:02.401242    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:07:05.569367 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:05.569307    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:07:05.890625 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:05.890524    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:07:12.401630 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:12.401583    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:07:15.568982 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:15.568932    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:07:15.890592 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:15.890495    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:07:22.401571 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:22.401514    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:07:25.568979 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:25.568921    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:07:25.890896 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:25.890801    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:07:32.401188 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:32.401085    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:07:35.569321 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:35.569256    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:07:35.889861 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:35.889762    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:07:42.401424 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:42.401382    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:07:45.568529 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:45.568477    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8001/health\": dial tcp 10.133.0.57:8001: connect: connection refused"
Jul 16 18:07:45.890389 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:45.890255    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:07:52.401134 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:52.401088    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:07:55.578909 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:55.578875    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:07:55.602172 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:55.602138    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:07:55.900558 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:55.900464    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:07:55.908682 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:07:55.908654    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:08:02.401573 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:02.401521    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:08:10.643433 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:10.643397    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"]
Jul 16 18:08:10.651195 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:10.651156    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"]
Jul 16 18:08:12.401104 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:12.401056    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:08:22.401247 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:22.401195    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8000/health\": dial tcp 10.133.0.59:8000: connect: connection refused"
Jul 16 18:08:25.676181 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:25.676139    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" containerID="cri-o://68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca" gracePeriod=45
Jul 16 18:08:25.683672 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:25.683616    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main" containerID="cri-o://af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4" gracePeriod=45
Jul 16 18:08:25.901812 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:25.901762    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 16 18:08:26.801760 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:26.801720    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="llm-d-routing-sidecar" containerID="cri-o://bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8" gracePeriod=44
Jul 16 18:08:26.940981 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:26.940953    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:08:27.049107 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049073    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-dshm\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.049335 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.049389 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049354    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-home\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.049449 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049385    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-model-cache\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.049449 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tls-certs\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.049557 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9gnc\" (UniqueName: \"kubernetes.io/projected/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-kube-api-access-s9gnc\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.049650 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.049515    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tmp-dir\") pod \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\" (UID: \"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1\") "
Jul 16 18:08:27.050071 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.050035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-home" (OuterVolumeSpecName: "home") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.050337 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.050316    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.050448 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.050382    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.051872 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.051845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-dshm" (OuterVolumeSpecName: "dshm") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.052301 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.052137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-kube-api-access-s9gnc" (OuterVolumeSpecName: "kube-api-access-s9gnc") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "kube-api-access-s9gnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:08:27.052459 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.052430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:08:27.052603 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.052578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f" (OuterVolumeSpecName: "kserve-pvc-source") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "pvc-816e24b0-2442-4458-a346-0c49c6eaace7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 18:08:27.062766 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.062733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" (UID: "45871f3c-410a-4e86-8b6b-6a5f29f7a9e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.067813 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.067793    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:08:27.150899 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.150869    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-home\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151075 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.150998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151140 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151084    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-model-cache\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151140 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8thjn\" (UniqueName: \"kubernetes.io/projected/51b15921-a3fc-4f4b-9366-a3d88805ed03-kube-api-access-8thjn\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151253 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-dshm\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151253 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151210    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-tmp-dir\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151253 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b15921-a3fc-4f4b-9366-a3d88805ed03-tls-certs\") pod \"51b15921-a3fc-4f4b-9366-a3d88805ed03\" (UID: \"51b15921-a3fc-4f4b-9366-a3d88805ed03\") "
Jul 16 18:08:27.151793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151608    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9gnc\" (UniqueName: \"kubernetes.io/projected/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-kube-api-access-s9gnc\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.151793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151635    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.151793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151631    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-home" (OuterVolumeSpecName: "home") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.151793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151649    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.151793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151662    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.151793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151676    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.152191 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.151830    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-model-cache" (OuterVolumeSpecName: "model-cache") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.154457 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.154409    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b15921-a3fc-4f4b-9366-a3d88805ed03-kube-api-access-8thjn" (OuterVolumeSpecName: "kube-api-access-8thjn") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "kube-api-access-8thjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:08:27.154457 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.154446    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-dshm" (OuterVolumeSpecName: "dshm") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.154777 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.154754    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51b15921-a3fc-4f4b-9366-a3d88805ed03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:08:27.154879 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.154863    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f" (OuterVolumeSpecName: "kserve-pvc-source") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "pvc-816e24b0-2442-4458-a346-0c49c6eaace7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 18:08:27.170728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.170697    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51b15921-a3fc-4f4b-9366-a3d88805ed03" (UID: "51b15921-a3fc-4f4b-9366-a3d88805ed03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:08:27.243324 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243201    2579 generic.go:356] "Generic (PLEG): container finished" podID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerID="af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4" exitCode=0
Jul 16 18:08:27.243324 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243232    2579 generic.go:356] "Generic (PLEG): container finished" podID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerID="bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8" exitCode=0
Jul 16 18:08:27.243324 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243258    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"
Jul 16 18:08:27.243324 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243250    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" event={"ID":"51b15921-a3fc-4f4b-9366-a3d88805ed03","Type":"ContainerDied","Data":"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"}
Jul 16 18:08:27.243684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243362    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" event={"ID":"51b15921-a3fc-4f4b-9366-a3d88805ed03","Type":"ContainerDied","Data":"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"}
Jul 16 18:08:27.243684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243393    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf" event={"ID":"51b15921-a3fc-4f4b-9366-a3d88805ed03","Type":"ContainerDied","Data":"61fd47b4eb535e904fe5000caed0ce00c6bc5ee1698a7ea707252f8cec3643ca"}
Jul 16 18:08:27.243684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.243403    2579 scope.go:117] "RemoveContainer" containerID="af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"
Jul 16 18:08:27.244824 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.244802    2579 generic.go:356] "Generic (PLEG): container finished" podID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerID="68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca" exitCode=0
Jul 16 18:08:27.244931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.244876    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"
Jul 16 18:08:27.244931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.244885    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" event={"ID":"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1","Type":"ContainerDied","Data":"68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca"}
Jul 16 18:08:27.245030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.244943    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn" event={"ID":"45871f3c-410a-4e86-8b6b-6a5f29f7a9e1","Type":"ContainerDied","Data":"8ada49568fb0d8bb7aadb4cf8f3723ffd28f6fa3bb0a9030ba898a907b8f174e"}
Jul 16 18:08:27.252912 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.252889    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.253021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.252921    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51b15921-a3fc-4f4b-9366-a3d88805ed03-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.253021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.252937    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.253021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.252984    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" "
Jul 16 18:08:27.253021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.253001    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.253021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.253017    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thjn\" (UniqueName: \"kubernetes.io/projected/51b15921-a3fc-4f4b-9366-a3d88805ed03-kube-api-access-8thjn\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.253440 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.253030    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51b15921-a3fc-4f4b-9366-a3d88805ed03-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:08:27.254387 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.254370    2579 scope.go:117] "RemoveContainer" containerID="bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"
Jul 16 18:08:27.264040 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264009    2579 scope.go:117] "RemoveContainer" containerID="af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"
Jul 16 18:08:27.264344 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:08:27.264316    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4\": container with ID starting with af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4 not found: ID does not exist" containerID="af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"
Jul 16 18:08:27.264525 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264350    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"} err="failed to get container status \"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4\": rpc error: code = NotFound desc = could not find container \"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4\": container with ID starting with af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4 not found: ID does not exist"
Jul 16 18:08:27.264525 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264369    2579 scope.go:117] "RemoveContainer" containerID="bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"
Jul 16 18:08:27.264664 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:08:27.264652    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8\": container with ID starting with bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8 not found: ID does not exist" containerID="bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"
Jul 16 18:08:27.264747 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264672    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"} err="failed to get container status \"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8\": rpc error: code = NotFound desc = could not find container \"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8\": container with ID starting with bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8 not found: ID does not exist"
Jul 16 18:08:27.264747 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264686    2579 scope.go:117] "RemoveContainer" containerID="af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"
Jul 16 18:08:27.264941 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264922    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4"} err="failed to get container status \"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4\": rpc error: code = NotFound desc = could not find container \"af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4\": container with ID starting with af900fe4ef4578cfb759b507a3568b99441b54b73ea4fbd5bd5f6e7e01a61ca4 not found: ID does not exist"
Jul 16 18:08:27.265015 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.264941    2579 scope.go:117] "RemoveContainer" containerID="bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"
Jul 16 18:08:27.265198 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.265175    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8"} err="failed to get container status \"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8\": rpc error: code = NotFound desc = could not find container \"bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8\": container with ID starting with bcb27495c6c10ec6f3c7bcfdd130804bb32869b12c3f8a7dc405337c5bd24ab8 not found: ID does not exist"
Jul 16 18:08:27.265198 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.265197    2579 scope.go:117] "RemoveContainer" containerID="68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca"
Jul 16 18:08:27.267044 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.266960    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"]
Jul 16 18:08:27.271875 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.271852    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-79t95rn"]
Jul 16 18:08:27.276452 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.276427    2579 scope.go:117] "RemoveContainer" containerID="68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca"
Jul 16 18:08:27.276791 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:08:27.276766    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca\": container with ID starting with 68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca not found: ID does not exist" containerID="68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca"
Jul 16 18:08:27.276882 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.276799    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca"} err="failed to get container status \"68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca\": rpc error: code = NotFound desc = could not find container \"68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca\": container with ID starting with 68e50cd130cfdafa9eab5bddc9b521ee2013dd06fb885b14bda83fa93d61b1ca not found: ID does not exist"
Jul 16 18:08:27.287115 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.287091    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"]
Jul 16 18:08:27.288027 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.288012    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-816e24b0-2442-4458-a346-0c49c6eaace7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f") on node "ip-10-0-132-31.ec2.internal"
Jul 16 18:08:27.290915 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.290894    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-bbf66b468-wqjwf"]
Jul 16 18:08:27.354084 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:27.354048    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\""
Jul 16 18:08:29.103688 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:29.103652    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" path="/var/lib/kubelet/pods/45871f3c-410a-4e86-8b6b-6a5f29f7a9e1/volumes"
Jul 16 18:08:29.104084 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:29.104070    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" path="/var/lib/kubelet/pods/51b15921-a3fc-4f4b-9366-a3d88805ed03/volumes"
Jul 16 18:08:32.410917 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:32.410879    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:08:32.418714 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:32.418690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:08:33.397954 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.397916    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 18:08:33.398422 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398398    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main"
Jul 16 18:08:33.398422 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398420    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398445    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="llm-d-routing-sidecar"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398453    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="llm-d-routing-sidecar"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398476    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398484    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398579    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="llm-d-routing-sidecar"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398591    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="45871f3c-410a-4e86-8b6b-6a5f29f7a9e1" containerName="main"
Jul 16 18:08:33.398622 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.398600    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="51b15921-a3fc-4f4b-9366-a3d88805ed03" containerName="main"
Jul 16 18:08:33.404017 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.403998    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.408053 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.408024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 16 18:08:33.408174 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.408143    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-r5srj\""
Jul 16 18:08:33.417615 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.417588    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 18:08:33.500656 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.500624    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 18:08:33.504434 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.504408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.517058 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.517035    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6jt6\" (UniqueName: \"kubernetes.io/projected/a772de12-a81e-47e9-992c-0b42494af7b3-kube-api-access-l6jt6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.517218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.517071    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a772de12-a81e-47e9-992c-0b42494af7b3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.517218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.517122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.517218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.517150    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.517218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.517183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.517442 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.517314    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.528874 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.528843    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 18:08:33.618039 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618002    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.618039 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618038    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scclh\" (UniqueName: \"kubernetes.io/projected/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-kube-api-access-scclh\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.618290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6jt6\" (UniqueName: \"kubernetes.io/projected/a772de12-a81e-47e9-992c-0b42494af7b3-kube-api-access-l6jt6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a772de12-a81e-47e9-992c-0b42494af7b3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618112    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.618290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618144    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618341    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.618563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618404    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.618563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618441    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618518    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.618563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618550    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618628    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.618792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.618740    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.620423 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.620406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.620617 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.620598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a772de12-a81e-47e9-992c-0b42494af7b3-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.642063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.641995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6jt6\" (UniqueName: \"kubernetes.io/projected/a772de12-a81e-47e9-992c-0b42494af7b3-kube-api-access-l6jt6\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:33.719501 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.719501 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719501    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scclh\" (UniqueName: \"kubernetes.io/projected/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-kube-api-access-scclh\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.719751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.719751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719571    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.719751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.719751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719625    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.719975 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.719957    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.720040 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.720005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.720040 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.720026    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.721900 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.721872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.722072 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.722056    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:33.730432 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:33.730409    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scclh\" (UniqueName: \"kubernetes.io/projected/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-kube-api-access-scclh\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.381655 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.381620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.390475 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.390444    2579 operation_generator.go:1470] "Controller attach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.483206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.483169    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.483404 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.483344    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.487495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.487457    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.569915 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.569885    2579 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/b45f7228e14240b65ad12bdfed958115c57791e355291dc13fb10bdd9eb08268/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.582937 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.582906    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.685535 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.685436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.685695 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.685577    2579 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.689646 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.689620    2579 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.702070 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.702044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.720728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.720705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-r5srj\""
Jul 16 18:08:39.728333 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.728304    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:39.815671 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.815640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:08:39.879537 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.879369    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 18:08:39.882939 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:08:39.882899    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda772de12_a81e_47e9_992c_0b42494af7b3.slice/crio-4c705d7704a672dcde5f41fa243b182ff69346b932ff9aaf091f36b53810639c WatchSource:0}: Error finding container 4c705d7704a672dcde5f41fa243b182ff69346b932ff9aaf091f36b53810639c: Status 404 returned error can't find the container with id 4c705d7704a672dcde5f41fa243b182ff69346b932ff9aaf091f36b53810639c
Jul 16 18:08:39.975632 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:39.975598    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 18:08:39.975943 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:08:39.975915    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dabe8e8_9a80_4059_b26d_2abd0ca0c038.slice/crio-4a4667d5b759f8597260f17159d8937102f2094d5dd5e85801507c94ad9af778 WatchSource:0}: Error finding container 4a4667d5b759f8597260f17159d8937102f2094d5dd5e85801507c94ad9af778: Status 404 returned error can't find the container with id 4a4667d5b759f8597260f17159d8937102f2094d5dd5e85801507c94ad9af778
Jul 16 18:08:40.328854 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.328810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"4dabe8e8-9a80-4059-b26d-2abd0ca0c038","Type":"ContainerStarted","Data":"5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7"}
Jul 16 18:08:40.329074 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.328882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"4dabe8e8-9a80-4059-b26d-2abd0ca0c038","Type":"ContainerStarted","Data":"4a4667d5b759f8597260f17159d8937102f2094d5dd5e85801507c94ad9af778"}
Jul 16 18:08:40.330934 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.330903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a772de12-a81e-47e9-992c-0b42494af7b3","Type":"ContainerStarted","Data":"b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758"}
Jul 16 18:08:40.331075 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.330957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a772de12-a81e-47e9-992c-0b42494af7b3","Type":"ContainerStarted","Data":"4c705d7704a672dcde5f41fa243b182ff69346b932ff9aaf091f36b53810639c"}
Jul 16 18:08:40.368279 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.368222    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=7.3682039679999995 podStartE2EDuration="7.368203968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:08:40.365894106 +0000 UTC m=+2041.749364053" watchObservedRunningTime="2026-07-16 18:08:40.368203968 +0000 UTC m=+2041.751673902"
Jul 16 18:08:40.413042 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.412965    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=7.412943886 podStartE2EDuration="7.412943886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:08:40.411384859 +0000 UTC m=+2041.794854795" watchObservedRunningTime="2026-07-16 18:08:40.412943886 +0000 UTC m=+2041.796413821"
Jul 16 18:08:40.729577 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:40.729484    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:49.729398 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:49.729353    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:08:49.731257 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:49.731226    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:08:51.442448 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:51.442411    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"]
Jul 16 18:08:59.729487 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:08:59.729441    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:09:06.467667 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:06.467556    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main" containerID="cri-o://cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa" gracePeriod=45
Jul 16 18:09:07.248846 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.248817    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:09:07.407472 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407436    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqzwl\" (UniqueName: \"kubernetes.io/projected/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kube-api-access-sqzwl\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.407664 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407495    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-dshm\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.407664 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407529    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-model-cache\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.407759 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407734    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kserve-provision-location\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.407807 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:07.407807 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407772    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tls-certs\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.407916 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407864    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tmp-dir\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.407972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.407922    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-home\") pod \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\" (UID: \"dee5db9b-eef3-4f9b-a818-1d91d2f783a8\") "
Jul 16 18:09:07.408728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.408389    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.408728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.408686    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-home" (OuterVolumeSpecName: "home") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:07.410391 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.410354    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-dshm" (OuterVolumeSpecName: "dshm") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:07.410517 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.410455    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:09:07.410591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.410565    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kube-api-access-sqzwl" (OuterVolumeSpecName: "kube-api-access-sqzwl") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "kube-api-access-sqzwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:09:07.420125 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.420100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:07.483473 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.483420    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dee5db9b-eef3-4f9b-a818-1d91d2f783a8" (UID: "dee5db9b-eef3-4f9b-a818-1d91d2f783a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:09:07.507922 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.507885    2579 generic.go:356] "Generic (PLEG): container finished" podID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerID="cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa" exitCode=0
Jul 16 18:09:07.508112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.507939    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"
Jul 16 18:09:07.508112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.507965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" event={"ID":"dee5db9b-eef3-4f9b-a818-1d91d2f783a8","Type":"ContainerDied","Data":"cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa"}
Jul 16 18:09:07.508112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.508020    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx" event={"ID":"dee5db9b-eef3-4f9b-a818-1d91d2f783a8","Type":"ContainerDied","Data":"65b8543432416eb9674eee7f6699c0e64b2507a7107ed8d7bcca44f42788ab0c"}
Jul 16 18:09:07.508112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.508045    2579 scope.go:117] "RemoveContainer" containerID="cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa"
Jul 16 18:09:07.509029 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.509004    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.509029 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.509031    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.509211 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.509045    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqzwl\" (UniqueName: \"kubernetes.io/projected/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kube-api-access-sqzwl\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.509211 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.509060    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.509211 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.509074    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.509211 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.509089    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dee5db9b-eef3-4f9b-a818-1d91d2f783a8-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:09:07.518467 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.518445    2579 scope.go:117] "RemoveContainer" containerID="5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba"
Jul 16 18:09:07.527919 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.527897    2579 scope.go:117] "RemoveContainer" containerID="cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa"
Jul 16 18:09:07.528216 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:09:07.528193    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa\": container with ID starting with cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa not found: ID does not exist" containerID="cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa"
Jul 16 18:09:07.528322 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.528230    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa"} err="failed to get container status \"cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa\": rpc error: code = NotFound desc = could not find container \"cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa\": container with ID starting with cf9c8dd8a9e9d66696f8040830eb5ae5bfcf9f7b1fea2e8a7c28a0fc1a70e9fa not found: ID does not exist"
Jul 16 18:09:07.528322 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.528255    2579 scope.go:117] "RemoveContainer" containerID="5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba"
Jul 16 18:09:07.528592 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:09:07.528567    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba\": container with ID starting with 5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba not found: ID does not exist" containerID="5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba"
Jul 16 18:09:07.528644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.528600    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba"} err="failed to get container status \"5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba\": rpc error: code = NotFound desc = could not find container \"5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba\": container with ID starting with 5b85d063be37d702307f5b0a75de0770d32d1b4d44caf869261cafe2c360f6ba not found: ID does not exist"
Jul 16 18:09:07.540575 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.540543    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"]
Jul 16 18:09:07.550486 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:07.550461    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-d4b5cbd7-ntdnx"]
Jul 16 18:09:09.103486 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:09.103452    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" path="/var/lib/kubelet/pods/dee5db9b-eef3-4f9b-a818-1d91d2f783a8/volumes"
Jul 16 18:09:09.728914 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:09.728868    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:09:19.729641 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:19.729590    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:09:29.729204 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:29.729159    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:09:32.203987 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.203945    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"]
Jul 16 18:09:32.204611 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.204358    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="storage-initializer"
Jul 16 18:09:32.204611 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.204371    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="storage-initializer"
Jul 16 18:09:32.204611 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.204387    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main"
Jul 16 18:09:32.204611 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.204394    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main"
Jul 16 18:09:32.204611 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.204464    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="dee5db9b-eef3-4f9b-a818-1d91d2f783a8" containerName="main"
Jul 16 18:09:32.207725 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.207704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.210997 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.210975    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 16 18:09:32.211131 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.210982    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:09:32.226523 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.226486    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"]
Jul 16 18:09:32.362470 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-dshm\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.362662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nn5\" (UniqueName: \"kubernetes.io/projected/357309d5-da1c-45c6-850a-d7b49af9d236-kube-api-access-49nn5\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.362662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362561    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357309d5-da1c-45c6-850a-d7b49af9d236-tls-certs\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.362662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362603    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-model-cache\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.362662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362651    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-kserve-provision-location\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.362844 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-tmp-dir\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.362844 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.362732    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-home\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.463804 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.463698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357309d5-da1c-45c6-850a-d7b49af9d236-tls-certs\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.463804 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.463749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-model-cache\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.463804 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.463775    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-kserve-provision-location\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464085 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464001    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-tmp-dir\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464149 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464085    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-home\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464149 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-dshm\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464248 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nn5\" (UniqueName: \"kubernetes.io/projected/357309d5-da1c-45c6-850a-d7b49af9d236-kube-api-access-49nn5\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464345 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-kserve-provision-location\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464437 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-tmp-dir\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464497 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-home\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.464738 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.464709    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-model-cache\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.466407 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.466374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357309d5-da1c-45c6-850a-d7b49af9d236-tls-certs\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.466527 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.466408    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-dshm\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.474048 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.474022    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nn5\" (UniqueName: \"kubernetes.io/projected/357309d5-da1c-45c6-850a-d7b49af9d236-kube-api-access-49nn5\") pod \"router-with-refs-test-kserve-68cd5c496c-5htr4\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.518799 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.518753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:32.703256 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:32.703228    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"]
Jul 16 18:09:32.704745 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:09:32.704713    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357309d5_da1c_45c6_850a_d7b49af9d236.slice/crio-44bf25681991516d474c4cb76cf06b5af0ad5e15989775c85c84c8f019206b02 WatchSource:0}: Error finding container 44bf25681991516d474c4cb76cf06b5af0ad5e15989775c85c84c8f019206b02: Status 404 returned error can't find the container with id 44bf25681991516d474c4cb76cf06b5af0ad5e15989775c85c84c8f019206b02
Jul 16 18:09:33.687826 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:33.687787    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" event={"ID":"357309d5-da1c-45c6-850a-d7b49af9d236","Type":"ContainerStarted","Data":"4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7"}
Jul 16 18:09:33.688363 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:33.687850    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" event={"ID":"357309d5-da1c-45c6-850a-d7b49af9d236","Type":"ContainerStarted","Data":"44bf25681991516d474c4cb76cf06b5af0ad5e15989775c85c84c8f019206b02"}
Jul 16 18:09:36.706986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:36.706945    2579 generic.go:356] "Generic (PLEG): container finished" podID="357309d5-da1c-45c6-850a-d7b49af9d236" containerID="4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7" exitCode=0
Jul 16 18:09:36.707403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:36.707009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" event={"ID":"357309d5-da1c-45c6-850a-d7b49af9d236","Type":"ContainerDied","Data":"4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7"}
Jul 16 18:09:37.714972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:37.714929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" event={"ID":"357309d5-da1c-45c6-850a-d7b49af9d236","Type":"ContainerStarted","Data":"47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e"}
Jul 16 18:09:37.743915 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:37.743851    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podStartSLOduration=2.656372185 podStartE2EDuration="5.743830269s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.087458084s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:09:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:09:37.742750674 +0000 UTC m=+2099.126220606" watchObservedRunningTime="2026-07-16 18:09:37.743830269 +0000 UTC m=+2099.127300203"
Jul 16 18:09:38.519073 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:38.519033    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:39.729030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:39.728981    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:09:42.519881 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:42.519837    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:09:42.521701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:42.521669    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:09:48.282368 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:48.282338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:09:48.286890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:48.286867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:09:49.729606 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:49.729565    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:09:52.519992 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:52.519947    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:09:59.729815 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:09:59.729768    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:10:02.519650 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:02.519598    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:10:09.729802 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:09.729751    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:10:12.520216 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:12.520157    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:10:19.728771 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:19.728717    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:10:22.520098 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:22.520044    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:10:29.729672 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:29.729619    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:10:32.520207 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:32.520088    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:10:39.728907 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:39.728857    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:10:42.520070 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:42.520021    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:10:49.729393 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:49.729344    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:10:52.520242 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:52.520196    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:10:59.729114 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:10:59.729052    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:11:02.519569 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:02.519525    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:11:09.739570 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:09.739527    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:11:09.747887 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:09.747853    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:11:12.519732 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:12.519676    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:11:22.520058 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:22.520013    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:11:32.519521 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:32.519471    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:11:42.530115 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:42.530083    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:11:42.537865 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:42.537832    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:11:47.092376 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:11:47.092336    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"]
Jul 16 18:12:02.116701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.116578    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" containerID="cri-o://47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e" gracePeriod=45
Jul 16 18:12:02.531133 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.531020    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 16 18:12:02.877443 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.877418    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:12:02.951252 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951212    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-tmp-dir\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951252 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951275    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-model-cache\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951545 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951331    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-dshm\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951545 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951350    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-home\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951545 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-kserve-provision-location\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951545 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951465    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49nn5\" (UniqueName: \"kubernetes.io/projected/357309d5-da1c-45c6-850a-d7b49af9d236-kube-api-access-49nn5\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951545 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357309d5-da1c-45c6-850a-d7b49af9d236-tls-certs\") pod \"357309d5-da1c-45c6-850a-d7b49af9d236\" (UID: \"357309d5-da1c-45c6-850a-d7b49af9d236\") "
Jul 16 18:12:02.951794 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951632    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-model-cache" (OuterVolumeSpecName: "model-cache") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:02.951934 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.951909    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:02.952085 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.952057    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-home" (OuterVolumeSpecName: "home") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:02.953578 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.953551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-dshm" (OuterVolumeSpecName: "dshm") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:02.954160 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.954137    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/357309d5-da1c-45c6-850a-d7b49af9d236-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:12:02.954160 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.954141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357309d5-da1c-45c6-850a-d7b49af9d236-kube-api-access-49nn5" (OuterVolumeSpecName: "kube-api-access-49nn5") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "kube-api-access-49nn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:12:02.963795 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:02.963762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:03.009213 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.009169    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "357309d5-da1c-45c6-850a-d7b49af9d236" (UID: "357309d5-da1c-45c6-850a-d7b49af9d236"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:03.053317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.053287    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49nn5\" (UniqueName: \"kubernetes.io/projected/357309d5-da1c-45c6-850a-d7b49af9d236-kube-api-access-49nn5\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.053317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.053316    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/357309d5-da1c-45c6-850a-d7b49af9d236-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.053519 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.053354    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.053519 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.053366    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.053519 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.053374    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.053519 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.053384    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/357309d5-da1c-45c6-850a-d7b49af9d236-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:03.627032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.626996    2579 generic.go:356] "Generic (PLEG): container finished" podID="357309d5-da1c-45c6-850a-d7b49af9d236" containerID="47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e" exitCode=0
Jul 16 18:12:03.627578 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.627059    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"
Jul 16 18:12:03.627578 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.627081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" event={"ID":"357309d5-da1c-45c6-850a-d7b49af9d236","Type":"ContainerDied","Data":"47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e"}
Jul 16 18:12:03.627578 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.627140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4" event={"ID":"357309d5-da1c-45c6-850a-d7b49af9d236","Type":"ContainerDied","Data":"44bf25681991516d474c4cb76cf06b5af0ad5e15989775c85c84c8f019206b02"}
Jul 16 18:12:03.627578 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.627162    2579 scope.go:117] "RemoveContainer" containerID="47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e"
Jul 16 18:12:03.637563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.637514    2579 scope.go:117] "RemoveContainer" containerID="4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7"
Jul 16 18:12:03.648963 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.648930    2579 scope.go:117] "RemoveContainer" containerID="47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e"
Jul 16 18:12:03.649231 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:03.649211    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e\": container with ID starting with 47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e not found: ID does not exist" containerID="47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e"
Jul 16 18:12:03.649332 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.649240    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e"} err="failed to get container status \"47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e\": rpc error: code = NotFound desc = could not find container \"47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e\": container with ID starting with 47558b1ba9ff20484f322e538fc78895fb0e9177e4f9590210fd50ead00e1d9e not found: ID does not exist"
Jul 16 18:12:03.649332 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.649257    2579 scope.go:117] "RemoveContainer" containerID="4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7"
Jul 16 18:12:03.649522 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:03.649505    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7\": container with ID starting with 4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7 not found: ID does not exist" containerID="4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7"
Jul 16 18:12:03.649564 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.649527    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7"} err="failed to get container status \"4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7\": rpc error: code = NotFound desc = could not find container \"4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7\": container with ID starting with 4bb4eb356db4950b5813a2988aa714c08b9e3151a63793e303446ca499e466f7 not found: ID does not exist"
Jul 16 18:12:03.655122 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.655101    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"]
Jul 16 18:12:03.663714 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:03.663689    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-68cd5c496c-5htr4"]
Jul 16 18:12:05.104206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:05.104169    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" path="/var/lib/kubelet/pods/357309d5-da1c-45c6-850a-d7b49af9d236/volumes"
Jul 16 18:12:14.971449 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.971411    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"]
Jul 16 18:12:14.971870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.971793    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="storage-initializer"
Jul 16 18:12:14.971870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.971805    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="storage-initializer"
Jul 16 18:12:14.971870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.971814    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main"
Jul 16 18:12:14.971870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.971820    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main"
Jul 16 18:12:14.972012 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.971886    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="357309d5-da1c-45c6-850a-d7b49af9d236" containerName="main"
Jul 16 18:12:14.974691 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.974665    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:14.980191 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.980163    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:12:14.980410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.980389    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 16 18:12:14.980920 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.980903    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-8ngx6\""
Jul 16 18:12:14.992314 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.992288    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"]
Jul 16 18:12:14.995547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.995524    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:14.998636 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:14.998613    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"]
Jul 16 18:12:15.010351 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.010323    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"]
Jul 16 18:12:15.064537 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.064723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064577    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.064723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrz57\" (UniqueName: \"kubernetes.io/projected/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kube-api-access-hrz57\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.064723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064619    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720ac9c1-b8fd-488d-922e-2ea154afaf76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.064723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.064890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.064890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064744    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.064890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.064890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.065056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.065056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.065056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.064986    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fff86\" (UniqueName: \"kubernetes.io/projected/720ac9c1-b8fd-488d-922e-2ea154afaf76-kube-api-access-fff86\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.065056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.065029    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.065056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.065049    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.165856 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.165811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.165865    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.165918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.165946    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fff86\" (UniqueName: \"kubernetes.io/projected/720ac9c1-b8fd-488d-922e-2ea154afaf76-kube-api-access-fff86\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166313 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166313 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166099    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166313 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166313 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166345    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166353    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrz57\" (UniqueName: \"kubernetes.io/projected/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kube-api-access-hrz57\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720ac9c1-b8fd-488d-922e-2ea154afaf76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166495    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166702    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.166999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166792    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.166999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.166938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.168389 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.168360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.168747 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.168728    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.169034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.169005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.169139 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.169036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720ac9c1-b8fd-488d-922e-2ea154afaf76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.176915 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.176888    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fff86\" (UniqueName: \"kubernetes.io/projected/720ac9c1-b8fd-488d-922e-2ea154afaf76-kube-api-access-fff86\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.177118 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.177101    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrz57\" (UniqueName: \"kubernetes.io/projected/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kube-api-access-hrz57\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.287651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.287546    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.306945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.306922    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:15.451827 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.451792    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"]
Jul 16 18:12:15.452775 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:12:15.452746    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720ac9c1_b8fd_488d_922e_2ea154afaf76.slice/crio-bb64eadc3fabe1ba4003c1a6f7a531408443d465c3043d642605d27d326bb788 WatchSource:0}: Error finding container bb64eadc3fabe1ba4003c1a6f7a531408443d465c3043d642605d27d326bb788: Status 404 returned error can't find the container with id bb64eadc3fabe1ba4003c1a6f7a531408443d465c3043d642605d27d326bb788
Jul 16 18:12:15.469765 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.469731    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"]
Jul 16 18:12:15.471865 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:12:15.471843    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0693a1cf_6030_4ff6_ba16_f8755e3d7d74.slice/crio-28dcc8019eaee40cf9449ae606436bebf863d220acae7589d996c7b50d3dbe2b WatchSource:0}: Error finding container 28dcc8019eaee40cf9449ae606436bebf863d220acae7589d996c7b50d3dbe2b: Status 404 returned error can't find the container with id 28dcc8019eaee40cf9449ae606436bebf863d220acae7589d996c7b50d3dbe2b
Jul 16 18:12:15.694095 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.694059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerStarted","Data":"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"}
Jul 16 18:12:15.694320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.694121    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerStarted","Data":"bb64eadc3fabe1ba4003c1a6f7a531408443d465c3043d642605d27d326bb788"}
Jul 16 18:12:15.694320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.694239    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:15.695983 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.695937    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" event={"ID":"0693a1cf-6030-4ff6-ba16-f8755e3d7d74","Type":"ContainerStarted","Data":"c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060"}
Jul 16 18:12:15.696104 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:15.696011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" event={"ID":"0693a1cf-6030-4ff6-ba16-f8755e3d7d74","Type":"ContainerStarted","Data":"28dcc8019eaee40cf9449ae606436bebf863d220acae7589d996c7b50d3dbe2b"}
Jul 16 18:12:16.703964 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:16.703926    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerStarted","Data":"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"}
Jul 16 18:12:19.725931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:19.725891    2579 generic.go:356] "Generic (PLEG): container finished" podID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerID="6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990" exitCode=0
Jul 16 18:12:19.726451 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:19.725962    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerDied","Data":"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"}
Jul 16 18:12:19.728158 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:19.728014    2579 generic.go:356] "Generic (PLEG): container finished" podID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerID="c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060" exitCode=0
Jul 16 18:12:19.728158 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:19.728042    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" event={"ID":"0693a1cf-6030-4ff6-ba16-f8755e3d7d74","Type":"ContainerDied","Data":"c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060"}
Jul 16 18:12:20.737020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:20.736985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerStarted","Data":"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"}
Jul 16 18:12:20.739437 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:20.739405    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" event={"ID":"0693a1cf-6030-4ff6-ba16-f8755e3d7d74","Type":"ContainerStarted","Data":"879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be"}
Jul 16 18:12:20.792747 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:20.792681    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podStartSLOduration=3.575264716 podStartE2EDuration="6.792658434s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.217393718s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:12:20.789586703 +0000 UTC m=+2262.173056639" watchObservedRunningTime="2026-07-16 18:12:20.792658434 +0000 UTC m=+2262.176128360"
Jul 16 18:12:20.857514 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:20.857444    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podStartSLOduration=3.490358267 podStartE2EDuration="6.857420928s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.367062661s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:12:20.849668984 +0000 UTC m=+2262.233138918" watchObservedRunningTime="2026-07-16 18:12:20.857420928 +0000 UTC m=+2262.240890863"
Jul 16 18:12:25.288367 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.288312    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:25.288882 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.288395    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:25.290491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.290194    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:12:25.307199 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.307166    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:25.307672 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.307647    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:12:25.307813 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.307759    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:12:25.309332 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:25.309298    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:12:31.379378 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:31.379344    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 18:12:34.097464 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:34.097423    2579 kubelet_pods.go:1085] "Unable to retrieve pull secret, the image pull may not succeed." pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" secret="" err="secret \"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-r5srj\" not found"
Jul 16 18:12:34.168945 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:34.168897    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:34.169133 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:34.168996    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs podName:4dabe8e8-9a80-4059-b26d-2abd0ca0c038 nodeName:}" failed. No retries permitted until 2026-07-16 18:12:34.668974198 +0000 UTC m=+2276.052444124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:34.675483 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:34.675434    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:34.675689 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:34.675546    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs podName:4dabe8e8-9a80-4059-b26d-2abd0ca0c038 nodeName:}" failed. No retries permitted until 2026-07-16 18:12:35.675525566 +0000 UTC m=+2277.058995498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:35.288215 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:35.288165    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:12:35.308566 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:35.308521    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:12:35.687644 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:35.687609    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:35.687834 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:35.687682    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs podName:4dabe8e8-9a80-4059-b26d-2abd0ca0c038 nodeName:}" failed. No retries permitted until 2026-07-16 18:12:37.687667189 +0000 UTC m=+2279.071137101 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:37.712093 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:37.712049    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:37.712532 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:37.712161    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs podName:4dabe8e8-9a80-4059-b26d-2abd0ca0c038 nodeName:}" failed. No retries permitted until 2026-07-16 18:12:41.712140206 +0000 UTC m=+2283.095610122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:41.755708 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:41.755670    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:41.756099 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:41.755798    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs podName:4dabe8e8-9a80-4059-b26d-2abd0ca0c038 nodeName:}" failed. No retries permitted until 2026-07-16 18:12:49.755776565 +0000 UTC m=+2291.139246497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:45.289098 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:45.289041    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:12:45.307399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:45.307361    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:12:46.405552 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:46.405503    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" containerID="cri-o://b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758" gracePeriod=45
Jul 16 18:12:46.740644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:46.740531    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 16 18:12:47.237173 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.237139    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:12:47.422231 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422231 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422251    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-dshm\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422285    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-home\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-model-cache\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422346    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-tmp-dir\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6jt6\" (UniqueName: \"kubernetes.io/projected/a772de12-a81e-47e9-992c-0b42494af7b3-kube-api-access-l6jt6\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422416    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a772de12-a81e-47e9-992c-0b42494af7b3-tls-certs\") pod \"a772de12-a81e-47e9-992c-0b42494af7b3\" (UID: \"a772de12-a81e-47e9-992c-0b42494af7b3\") "
Jul 16 18:12:47.422821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422599    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:47.423158 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.422825    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.423380 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.423346    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-home" (OuterVolumeSpecName: "home") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:47.425074 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.425033    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-dshm" (OuterVolumeSpecName: "dshm") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:47.425803 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.425771    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a772de12-a81e-47e9-992c-0b42494af7b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:12:47.425913 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.425877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a772de12-a81e-47e9-992c-0b42494af7b3-kube-api-access-l6jt6" (OuterVolumeSpecName: "kube-api-access-l6jt6") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "kube-api-access-l6jt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:12:47.426329 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.426308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f" (OuterVolumeSpecName: "kserve-pvc-source") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "pvc-816e24b0-2442-4458-a346-0c49c6eaace7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 18:12:47.434139 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.434110    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a772de12-a81e-47e9-992c-0b42494af7b3" (UID: "a772de12-a81e-47e9-992c-0b42494af7b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:12:47.523761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.523719    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.523761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.523752    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6jt6\" (UniqueName: \"kubernetes.io/projected/a772de12-a81e-47e9-992c-0b42494af7b3-kube-api-access-l6jt6\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.523761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.523764    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a772de12-a81e-47e9-992c-0b42494af7b3-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.523761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.523773    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.523761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.523780    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a772de12-a81e-47e9-992c-0b42494af7b3-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:12:47.930121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.930078    2579 generic.go:356] "Generic (PLEG): container finished" podID="a772de12-a81e-47e9-992c-0b42494af7b3" containerID="b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758" exitCode=0
Jul 16 18:12:47.930317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.930132    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 16 18:12:47.930317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.930156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a772de12-a81e-47e9-992c-0b42494af7b3","Type":"ContainerDied","Data":"b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758"}
Jul 16 18:12:47.930317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.930212    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"a772de12-a81e-47e9-992c-0b42494af7b3","Type":"ContainerDied","Data":"4c705d7704a672dcde5f41fa243b182ff69346b932ff9aaf091f36b53810639c"}
Jul 16 18:12:47.930317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.930237    2579 scope.go:117] "RemoveContainer" containerID="b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758"
Jul 16 18:12:47.939775 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.939755    2579 scope.go:117] "RemoveContainer" containerID="b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758"
Jul 16 18:12:47.940133 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:47.940111    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758\": container with ID starting with b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758 not found: ID does not exist" containerID="b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758"
Jul 16 18:12:47.940213 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.940145    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758"} err="failed to get container status \"b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758\": rpc error: code = NotFound desc = could not find container \"b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758\": container with ID starting with b47018a1774dfc43d518798188899cd1397adbfb80cfcbc15da450f2e2edc758 not found: ID does not exist"
Jul 16 18:12:47.965355 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.960070    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 18:12:47.968226 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.968084    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 16 18:12:47.985321 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:47.985258    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 18:12:49.104260 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:49.104227    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" path="/var/lib/kubelet/pods/a772de12-a81e-47e9-992c-0b42494af7b3/volumes"
Jul 16 18:12:49.849155 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:49.849122    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs: secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:49.849350 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:12:49.849193    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs podName:4dabe8e8-9a80-4059-b26d-2abd0ca0c038 nodeName:}" failed. No retries permitted until 2026-07-16 18:13:05.849177828 +0000 UTC m=+2307.232647741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs") pod "llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038") : secret "llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs" not found
Jul 16 18:12:55.288632 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:55.288581    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:12:55.308050 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:12:55.308006    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:13:03.009023 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.008967    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="4dabe8e8-9a80-4059-b26d-2abd0ca0c038" containerName="main" containerID="cri-o://5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7" gracePeriod=45
Jul 16 18:13:03.798698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.798659    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:13:03.893701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893598    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.893701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.893942 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893722    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-home\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.893942 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893805    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scclh\" (UniqueName: \"kubernetes.io/projected/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-kube-api-access-scclh\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.893942 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893846    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tmp-dir\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.893942 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-dshm\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.893942 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.893938    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-model-cache\") pod \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\" (UID: \"4dabe8e8-9a80-4059-b26d-2abd0ca0c038\") "
Jul 16 18:13:03.894522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.894387    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-home" (OuterVolumeSpecName: "home") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:13:03.894708 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.894472    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-model-cache" (OuterVolumeSpecName: "model-cache") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:13:03.897010 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.896581    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:13:03.897010 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.896654    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-dshm" (OuterVolumeSpecName: "dshm") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:13:03.897010 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.896739    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-kube-api-access-scclh" (OuterVolumeSpecName: "kube-api-access-scclh") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "kube-api-access-scclh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:13:03.897809 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.897783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f" (OuterVolumeSpecName: "kserve-pvc-source") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "pvc-816e24b0-2442-4458-a346-0c49c6eaace7". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 16 18:13:03.905875 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.905845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4dabe8e8-9a80-4059-b26d-2abd0ca0c038" (UID: "4dabe8e8-9a80-4059-b26d-2abd0ca0c038"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:13:03.995279 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995235    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:13:03.995478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995292    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:13:03.995478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995341    2579 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" "
Jul 16 18:13:03.995478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995354    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:13:03.995478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995369    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:13:03.995478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995381    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scclh\" (UniqueName: \"kubernetes.io/projected/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-kube-api-access-scclh\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:13:03.995478 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:03.995392    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4dabe8e8-9a80-4059-b26d-2abd0ca0c038-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:13:04.025293 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.025251    2579 operation_generator.go:896] UnmountDevice succeeded for volume "pvc-816e24b0-2442-4458-a346-0c49c6eaace7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f") on node "ip-10-0-132-31.ec2.internal"
Jul 16 18:13:04.041875 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.041837    2579 generic.go:356] "Generic (PLEG): container finished" podID="4dabe8e8-9a80-4059-b26d-2abd0ca0c038" containerID="5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7" exitCode=0
Jul 16 18:13:04.042056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.041906    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 16 18:13:04.042056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.041929    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"4dabe8e8-9a80-4059-b26d-2abd0ca0c038","Type":"ContainerDied","Data":"5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7"}
Jul 16 18:13:04.042056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.041995    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"4dabe8e8-9a80-4059-b26d-2abd0ca0c038","Type":"ContainerDied","Data":"4a4667d5b759f8597260f17159d8937102f2094d5dd5e85801507c94ad9af778"}
Jul 16 18:13:04.042056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.042028    2579 scope.go:117] "RemoveContainer" containerID="5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7"
Jul 16 18:13:04.052662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.052634    2579 scope.go:117] "RemoveContainer" containerID="5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7"
Jul 16 18:13:04.053624 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:13:04.053605    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7\": container with ID starting with 5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7 not found: ID does not exist" containerID="5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7"
Jul 16 18:13:04.053717 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.053631    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7"} err="failed to get container status \"5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7\": rpc error: code = NotFound desc = could not find container \"5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7\": container with ID starting with 5df250141cb29b63c1c98125710961b2da5c63a005774d6665197e1a2ca489a7 not found: ID does not exist"
Jul 16 18:13:04.076353 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.076314    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 18:13:04.086134 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.086103    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 16 18:13:04.096547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:04.096516    2579 reconciler_common.go:299] "Volume detached for volume \"pvc-816e24b0-2442-4458-a346-0c49c6eaace7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e5b3d6b8d187f70f\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"csi-a046971e6ce85d493280c8d634056390757542c765b60f84e9b570a1a7fd9dcc\""
Jul 16 18:13:05.105688 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:05.105651    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dabe8e8-9a80-4059-b26d-2abd0ca0c038" path="/var/lib/kubelet/pods/4dabe8e8-9a80-4059-b26d-2abd0ca0c038/volumes"
Jul 16 18:13:05.288420 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:05.288374    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:13:05.307732 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:05.307684    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:13:11.837242 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837199    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"]
Jul 16 18:13:11.837817 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837797    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main"
Jul 16 18:13:11.837817 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837817    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main"
Jul 16 18:13:11.837966 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837867    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dabe8e8-9a80-4059-b26d-2abd0ca0c038" containerName="main"
Jul 16 18:13:11.837966 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837876    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dabe8e8-9a80-4059-b26d-2abd0ca0c038" containerName="main"
Jul 16 18:13:11.838083 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837970    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dabe8e8-9a80-4059-b26d-2abd0ca0c038" containerName="main"
Jul 16 18:13:11.838083 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.837988    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a772de12-a81e-47e9-992c-0b42494af7b3" containerName="main"
Jul 16 18:13:11.841464 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.841436    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.845218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.845190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 16 18:13:11.870072 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.870019    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"]
Jul 16 18:13:11.975670 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975620    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-model-cache\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.975852 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975707    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-home\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.975852 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd625de1-9cf8-4862-b1fa-504475b94c65-tls-certs\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.975852 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-dshm\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.975852 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975806    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-kserve-provision-location\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.975997 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975926    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llflb\" (UniqueName: \"kubernetes.io/projected/cd625de1-9cf8-4862-b1fa-504475b94c65-kube-api-access-llflb\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:11.975997 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:11.975963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-tmp-dir\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.076848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.076809    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-tmp-dir\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.076882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-model-cache\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.076919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-home\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.076938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd625de1-9cf8-4862-b1fa-504475b94c65-tls-certs\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.076957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-dshm\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.076980    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-kserve-provision-location\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077312 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.077049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llflb\" (UniqueName: \"kubernetes.io/projected/cd625de1-9cf8-4862-b1fa-504475b94c65-kube-api-access-llflb\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.077374    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-tmp-dir\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077551 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.077520    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-home\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.077597    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-kserve-provision-location\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.077829 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.077721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-model-cache\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.079575 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.079547    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-dshm\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.079833 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.079813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd625de1-9cf8-4862-b1fa-504475b94c65-tls-certs\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.087720 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.087626    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llflb\" (UniqueName: \"kubernetes.io/projected/cd625de1-9cf8-4862-b1fa-504475b94c65-kube-api-access-llflb\") pod \"stop-feature-test-kserve-f98bfbff-xltsj\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.153393 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.153353    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:12.328177 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:12.328138    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"]
Jul 16 18:13:12.329540 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:13:12.329505    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd625de1_9cf8_4862_b1fa_504475b94c65.slice/crio-f63d84625a2eca1c988ad23530f01d6b5277f567e64ae013eed96f20deb0ec1e WatchSource:0}: Error finding container f63d84625a2eca1c988ad23530f01d6b5277f567e64ae013eed96f20deb0ec1e: Status 404 returned error can't find the container with id f63d84625a2eca1c988ad23530f01d6b5277f567e64ae013eed96f20deb0ec1e
Jul 16 18:13:13.127203 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:13.127156    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" event={"ID":"cd625de1-9cf8-4862-b1fa-504475b94c65","Type":"ContainerStarted","Data":"da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e"}
Jul 16 18:13:13.127838 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:13.127808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" event={"ID":"cd625de1-9cf8-4862-b1fa-504475b94c65","Type":"ContainerStarted","Data":"f63d84625a2eca1c988ad23530f01d6b5277f567e64ae013eed96f20deb0ec1e"}
Jul 16 18:13:15.288782 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:15.288728    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:13:15.308221 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:15.308167    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:13:16.152428 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:16.152392    2579 generic.go:356] "Generic (PLEG): container finished" podID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerID="da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e" exitCode=0
Jul 16 18:13:16.152679 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:16.152462    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" event={"ID":"cd625de1-9cf8-4862-b1fa-504475b94c65","Type":"ContainerDied","Data":"da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e"}
Jul 16 18:13:17.164040 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:17.163990    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" event={"ID":"cd625de1-9cf8-4862-b1fa-504475b94c65","Type":"ContainerStarted","Data":"339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab"}
Jul 16 18:13:17.205888 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:17.205807    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podStartSLOduration=3.040199223 podStartE2EDuration="6.20578038s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.165581157s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:13:17.199961526 +0000 UTC m=+2318.583431461" watchObservedRunningTime="2026-07-16 18:13:17.20578038 +0000 UTC m=+2318.589250317"
Jul 16 18:13:18.153807 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:18.153757    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:22.154011 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:22.153960    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:13:22.156224 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:22.156181    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:13:25.288510 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:25.288456    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:13:25.308453 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:25.308406    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:13:32.154325 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:32.154200    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:13:35.288576 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:35.288519    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:13:35.307595 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:35.307536    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:13:42.154141 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:42.154093    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:13:45.288712 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:45.288667    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:13:45.308176 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:45.308129    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:13:52.154744 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:52.154692    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:13:55.288908 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:55.288855    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:13:55.308465 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:13:55.308425    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:14:02.154422 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:02.154371    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:14:05.289020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:05.288970    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:14:05.307913 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:05.307866    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:14:12.154113 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:12.154064    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:14:15.288841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:15.288793    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:14:15.307887 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:15.307851    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:14:22.154547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:22.154489    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:14:25.288105 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:25.288043    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:14:25.307526 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:25.307486    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:14:32.154321 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:32.154253    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:14:35.288620 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:35.288569    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:14:35.308132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:35.308090    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:14:42.154321 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:42.154253    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:14:45.288754 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:45.288700    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:14:45.307498 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:45.307451    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:14:48.326424 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:48.326382    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:14:48.330798 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:48.330774    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:14:52.153873 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:52.153825    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:14:55.288179 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:55.288085    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:14:55.308975 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:14:55.308930    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:15:02.154133 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:02.154036    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:15:05.288476 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:05.288426    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:15:05.308150 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:05.308112    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:15:12.154129 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:12.154082    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:15:15.288381 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:15.288322    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8001/health\": dial tcp 10.133.0.63:8001: connect: connection refused"
Jul 16 18:15:15.308336 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:15.308302    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 16 18:15:22.153883 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:22.153836    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:15:25.304697 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:25.304656    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:15:25.318059 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:25.318026    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:15:25.318723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:25.318701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:15:25.326468 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:25.326445    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:15:32.154414 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:32.154371    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:15:40.563043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:40.562989    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"]
Jul 16 18:15:40.565546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:40.565517    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"]
Jul 16 18:15:42.154624 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:42.154582    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8000/health\": dial tcp 10.133.0.65:8000: connect: connection refused"
Jul 16 18:15:52.164618 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:52.164584    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:15:52.173885 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:52.173850    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:15:53.767355 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:53.767309    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"]
Jul 16 18:15:55.587031 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:55.586992    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main" containerID="cri-o://da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1" gracePeriod=45
Jul 16 18:15:55.588947 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:55.588912    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main" containerID="cri-o://879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be" gracePeriod=45
Jul 16 18:15:56.414557 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.414510    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="llm-d-routing-sidecar" containerID="cri-o://66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd" gracePeriod=45
Jul 16 18:15:56.566211 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.566178    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:15:56.673063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673039    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:15:56.673420 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673326    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-home\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673420 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673407    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tmp-dir\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673441    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kserve-provision-location\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-dshm\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673595 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-model-cache\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673595 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673585    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tls-certs\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673694 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673614    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrz57\" (UniqueName: \"kubernetes.io/projected/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kube-api-access-hrz57\") pod \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\" (UID: \"0693a1cf-6030-4ff6-ba16-f8755e3d7d74\") "
Jul 16 18:15:56.673993 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673819    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-model-cache" (OuterVolumeSpecName: "model-cache") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.673993 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.673935    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.674165 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.674011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-home" (OuterVolumeSpecName: "home") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.675836 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.675814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:15:56.676052 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.676013    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kube-api-access-hrz57" (OuterVolumeSpecName: "kube-api-access-hrz57") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "kube-api-access-hrz57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:15:56.676143 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.676117    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-dshm" (OuterVolumeSpecName: "dshm") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.685866 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.685840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.746030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.745970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0693a1cf-6030-4ff6-ba16-f8755e3d7d74" (UID: "0693a1cf-6030-4ff6-ba16-f8755e3d7d74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.774692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-kserve-provision-location\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.774877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774742    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-home\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.774877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774764    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-model-cache\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.774877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774798    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720ac9c1-b8fd-488d-922e-2ea154afaf76-tls-certs\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.774877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774837    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fff86\" (UniqueName: \"kubernetes.io/projected/720ac9c1-b8fd-488d-922e-2ea154afaf76-kube-api-access-fff86\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.775106 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774911    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-tmp-dir\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.775106 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.774954    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-dshm\") pod \"720ac9c1-b8fd-488d-922e-2ea154afaf76\" (UID: \"720ac9c1-b8fd-488d-922e-2ea154afaf76\") "
Jul 16 18:15:56.775106 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-model-cache" (OuterVolumeSpecName: "model-cache") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.775425 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775346    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775425 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775372    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775425 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775387    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775425 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775403    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775454    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775470    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrz57\" (UniqueName: \"kubernetes.io/projected/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-kube-api-access-hrz57\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775482    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0693a1cf-6030-4ff6-ba16-f8755e3d7d74-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.775841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.775613    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-home" (OuterVolumeSpecName: "home") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.777893 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.777867    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-dshm" (OuterVolumeSpecName: "dshm") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.777893 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.777874    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720ac9c1-b8fd-488d-922e-2ea154afaf76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:15:56.778304 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.778259    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720ac9c1-b8fd-488d-922e-2ea154afaf76-kube-api-access-fff86" (OuterVolumeSpecName: "kube-api-access-fff86") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "kube-api-access-fff86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:15:56.787707 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.787668    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.835439 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.835393    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "720ac9c1-b8fd-488d-922e-2ea154afaf76" (UID: "720ac9c1-b8fd-488d-922e-2ea154afaf76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:15:56.876674 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.876640    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.876674 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.876669    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.876674 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.876680    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720ac9c1-b8fd-488d-922e-2ea154afaf76-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.876891 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.876690    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fff86\" (UniqueName: \"kubernetes.io/projected/720ac9c1-b8fd-488d-922e-2ea154afaf76-kube-api-access-fff86\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.876891 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.876698    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:56.876891 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:56.876706    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720ac9c1-b8fd-488d-922e-2ea154afaf76-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:15:57.209959 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.209920    2579 generic.go:356] "Generic (PLEG): container finished" podID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerID="da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1" exitCode=0
Jul 16 18:15:57.209959 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.209946    2579 generic.go:356] "Generic (PLEG): container finished" podID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerID="66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd" exitCode=0
Jul 16 18:15:57.210223 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.209969    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"
Jul 16 18:15:57.210223 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.210004    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerDied","Data":"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"}
Jul 16 18:15:57.210223 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.210064    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerDied","Data":"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"}
Jul 16 18:15:57.210223 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.210083    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp" event={"ID":"720ac9c1-b8fd-488d-922e-2ea154afaf76","Type":"ContainerDied","Data":"bb64eadc3fabe1ba4003c1a6f7a531408443d465c3043d642605d27d326bb788"}
Jul 16 18:15:57.210223 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.210105    2579 scope.go:117] "RemoveContainer" containerID="da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"
Jul 16 18:15:57.211850 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.211820    2579 generic.go:356] "Generic (PLEG): container finished" podID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerID="879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be" exitCode=0
Jul 16 18:15:57.211997 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.211878    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"
Jul 16 18:15:57.211997 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.211893    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" event={"ID":"0693a1cf-6030-4ff6-ba16-f8755e3d7d74","Type":"ContainerDied","Data":"879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be"}
Jul 16 18:15:57.211997 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.211935    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz" event={"ID":"0693a1cf-6030-4ff6-ba16-f8755e3d7d74","Type":"ContainerDied","Data":"28dcc8019eaee40cf9449ae606436bebf863d220acae7589d996c7b50d3dbe2b"}
Jul 16 18:15:57.219646 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.219628    2579 scope.go:117] "RemoveContainer" containerID="6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"
Jul 16 18:15:57.228313 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.228292    2579 scope.go:117] "RemoveContainer" containerID="66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"
Jul 16 18:15:57.237442 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.237415    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"]
Jul 16 18:15:57.239109 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239087    2579 scope.go:117] "RemoveContainer" containerID="da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"
Jul 16 18:15:57.239471 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:15:57.239445    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1\": container with ID starting with da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1 not found: ID does not exist" containerID="da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"
Jul 16 18:15:57.239560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239481    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"} err="failed to get container status \"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1\": rpc error: code = NotFound desc = could not find container \"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1\": container with ID starting with da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1 not found: ID does not exist"
Jul 16 18:15:57.239560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239499    2579 scope.go:117] "RemoveContainer" containerID="6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"
Jul 16 18:15:57.239762 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:15:57.239748    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990\": container with ID starting with 6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990 not found: ID does not exist" containerID="6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"
Jul 16 18:15:57.239803 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239766    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"} err="failed to get container status \"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990\": rpc error: code = NotFound desc = could not find container \"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990\": container with ID starting with 6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990 not found: ID does not exist"
Jul 16 18:15:57.239803 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239779    2579 scope.go:117] "RemoveContainer" containerID="66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"
Jul 16 18:15:57.239981 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:15:57.239966    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd\": container with ID starting with 66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd not found: ID does not exist" containerID="66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"
Jul 16 18:15:57.240030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239984    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"} err="failed to get container status \"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd\": rpc error: code = NotFound desc = could not find container \"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd\": container with ID starting with 66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd not found: ID does not exist"
Jul 16 18:15:57.240030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.239996    2579 scope.go:117] "RemoveContainer" containerID="da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"
Jul 16 18:15:57.240201 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.240185    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1"} err="failed to get container status \"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1\": rpc error: code = NotFound desc = could not find container \"da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1\": container with ID starting with da95c88a0e2f91f5c156afdea8ec3e0d57b0ac42ab1bd15aa5bffad1196aaac1 not found: ID does not exist"
Jul 16 18:15:57.240243 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.240202    2579 scope.go:117] "RemoveContainer" containerID="6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"
Jul 16 18:15:57.240458 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.240439    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990"} err="failed to get container status \"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990\": rpc error: code = NotFound desc = could not find container \"6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990\": container with ID starting with 6b5529a37dc8b55b6f8ff7769ed6a62ff3502ee3980a7416e069fe9a55509990 not found: ID does not exist"
Jul 16 18:15:57.240512 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.240457    2579 scope.go:117] "RemoveContainer" containerID="66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"
Jul 16 18:15:57.240692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.240674    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd"} err="failed to get container status \"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd\": rpc error: code = NotFound desc = could not find container \"66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd\": container with ID starting with 66f2cb5179b27bcbe648b8f4e7a09f5a5d19323aa62bc3d00aa2ff796b27f3dd not found: ID does not exist"
Jul 16 18:15:57.240743 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.240693    2579 scope.go:117] "RemoveContainer" containerID="879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be"
Jul 16 18:15:57.241291 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.241251    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-7cd67cfb9bhccbp"]
Jul 16 18:15:57.249141 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.249113    2579 scope.go:117] "RemoveContainer" containerID="c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060"
Jul 16 18:15:57.256192 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.256166    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"]
Jul 16 18:15:57.257987 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.257967    2579 scope.go:117] "RemoveContainer" containerID="879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be"
Jul 16 18:15:57.258300 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:15:57.258281    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be\": container with ID starting with 879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be not found: ID does not exist" containerID="879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be"
Jul 16 18:15:57.258361 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.258307    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be"} err="failed to get container status \"879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be\": rpc error: code = NotFound desc = could not find container \"879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be\": container with ID starting with 879d0cf78b1254f911eea8f4225d2b62616b55728b5ba2432cf7148ae4c6d4be not found: ID does not exist"
Jul 16 18:15:57.258361 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.258322    2579 scope.go:117] "RemoveContainer" containerID="c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060"
Jul 16 18:15:57.258578 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:15:57.258557    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060\": container with ID starting with c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060 not found: ID does not exist" containerID="c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060"
Jul 16 18:15:57.258652 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.258584    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060"} err="failed to get container status \"c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060\": rpc error: code = NotFound desc = could not find container \"c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060\": container with ID starting with c3b3bb966ce2ebce7b31df0679f01035ab1dfb4dd30b830e541bdb1c82cad060 not found: ID does not exist"
Jul 16 18:15:57.261627 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:57.261604    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-8cl8mwz"]
Jul 16 18:15:59.102990 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:59.102954    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" path="/var/lib/kubelet/pods/0693a1cf-6030-4ff6-ba16-f8755e3d7d74/volumes"
Jul 16 18:15:59.103449 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:15:59.103434    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" path="/var/lib/kubelet/pods/720ac9c1-b8fd-488d-922e-2ea154afaf76/volumes"
Jul 16 18:16:09.203949 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:09.203911    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main" containerID="cri-o://339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab" gracePeriod=45
Jul 16 18:16:09.969140 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:09.969114    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:16:10.101625 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101578    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-home\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.101625 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101626    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-kserve-provision-location\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.101908 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101661    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llflb\" (UniqueName: \"kubernetes.io/projected/cd625de1-9cf8-4862-b1fa-504475b94c65-kube-api-access-llflb\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.101908 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101687    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-dshm\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.101908 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101748    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd625de1-9cf8-4862-b1fa-504475b94c65-tls-certs\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.101908 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101765    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-tmp-dir\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.101908 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.101816    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-model-cache\") pod \"cd625de1-9cf8-4862-b1fa-504475b94c65\" (UID: \"cd625de1-9cf8-4862-b1fa-504475b94c65\") "
Jul 16 18:16:10.102229 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.102190    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:10.102446 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.102418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-home" (OuterVolumeSpecName: "home") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:10.103841 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.103817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-dshm" (OuterVolumeSpecName: "dshm") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:10.104259 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.104240    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd625de1-9cf8-4862-b1fa-504475b94c65-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:16:10.104259 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.104250    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd625de1-9cf8-4862-b1fa-504475b94c65-kube-api-access-llflb" (OuterVolumeSpecName: "kube-api-access-llflb") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "kube-api-access-llflb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:16:10.115588 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.115549    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:10.175758 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.175713    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd625de1-9cf8-4862-b1fa-504475b94c65" (UID: "cd625de1-9cf8-4862-b1fa-504475b94c65"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:16:10.202736 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202692    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd625de1-9cf8-4862-b1fa-504475b94c65-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.202736 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202734    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.202736 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202747    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.203009 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202759    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.203009 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202773    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.203009 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202786    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llflb\" (UniqueName: \"kubernetes.io/projected/cd625de1-9cf8-4862-b1fa-504475b94c65-kube-api-access-llflb\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.203009 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.202799    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd625de1-9cf8-4862-b1fa-504475b94c65-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:16:10.286561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.286527    2579 generic.go:356] "Generic (PLEG): container finished" podID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerID="339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab" exitCode=0
Jul 16 18:16:10.287021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.286572    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"
Jul 16 18:16:10.287021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.286611    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" event={"ID":"cd625de1-9cf8-4862-b1fa-504475b94c65","Type":"ContainerDied","Data":"339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab"}
Jul 16 18:16:10.287021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.286682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj" event={"ID":"cd625de1-9cf8-4862-b1fa-504475b94c65","Type":"ContainerDied","Data":"f63d84625a2eca1c988ad23530f01d6b5277f567e64ae013eed96f20deb0ec1e"}
Jul 16 18:16:10.287021 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.286719    2579 scope.go:117] "RemoveContainer" containerID="339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab"
Jul 16 18:16:10.296448 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.296430    2579 scope.go:117] "RemoveContainer" containerID="da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e"
Jul 16 18:16:10.305529 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.305513    2579 scope.go:117] "RemoveContainer" containerID="339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab"
Jul 16 18:16:10.305756 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:16:10.305741    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab\": container with ID starting with 339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab not found: ID does not exist" containerID="339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab"
Jul 16 18:16:10.305823 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.305765    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab"} err="failed to get container status \"339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab\": rpc error: code = NotFound desc = could not find container \"339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab\": container with ID starting with 339ba4e563d91fdfb330fb598a086a6d8d867460516156f99d0c417dad1188ab not found: ID does not exist"
Jul 16 18:16:10.305823 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.305780    2579 scope.go:117] "RemoveContainer" containerID="da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e"
Jul 16 18:16:10.305977 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:16:10.305963    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e\": container with ID starting with da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e not found: ID does not exist" containerID="da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e"
Jul 16 18:16:10.306018 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.305980    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e"} err="failed to get container status \"da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e\": rpc error: code = NotFound desc = could not find container \"da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e\": container with ID starting with da1f5bdeed67c64dd34877c6fd6b8ca512c7881c897cafe3b3462356da16947e not found: ID does not exist"
Jul 16 18:16:10.312440 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.312415    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"]
Jul 16 18:16:10.317696 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:10.317676    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-xltsj"]
Jul 16 18:16:11.103079 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:11.103046    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" path="/var/lib/kubelet/pods/cd625de1-9cf8-4862-b1fa-504475b94c65/volumes"
Jul 16 18:16:17.767483 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767444    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"]
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767813    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767824    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767832    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="storage-initializer"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767838    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="storage-initializer"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767851    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="storage-initializer"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767857    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="storage-initializer"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767870    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="llm-d-routing-sidecar"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767875    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="llm-d-routing-sidecar"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767883    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767888    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767894    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767899    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767907    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="storage-initializer"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767912    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="storage-initializer"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767969    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0693a1cf-6030-4ff6-ba16-f8755e3d7d74" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767976    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd625de1-9cf8-4862-b1fa-504475b94c65" containerName="main"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767983    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="llm-d-routing-sidecar"
Jul 16 18:16:17.767986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.767991    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="720ac9c1-b8fd-488d-922e-2ea154afaf76" containerName="main"
Jul 16 18:16:17.771068 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.771050    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.774061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.774024    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:16:17.774061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.774048    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:16:17.774259 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.774030    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:16:17.774259 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.774122    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 16 18:16:17.774259 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.774029    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-rtn5w\""
Jul 16 18:16:17.774259 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.774028    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:16:17.784314 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.784286    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"]
Jul 16 18:16:17.790624 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.790599    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"]
Jul 16 18:16:17.794220 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.794200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.809014 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.808987    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"]
Jul 16 18:16:17.869776 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869735    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.869962 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsm4g\" (UniqueName: \"kubernetes.io/projected/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kube-api-access-jsm4g\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.869962 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869847    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.869962 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-home\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.869962 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.869962 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-dshm\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.870137 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.869989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.971353 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971320    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431aaef4-6c9c-420b-9ddf-3e170d653e56-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.971561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.971561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971444    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-home\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.971561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971478    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.971561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.971561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971520    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg642\" (UniqueName: \"kubernetes.io/projected/431aaef4-6c9c-420b-9ddf-3e170d653e56-kube-api-access-mg642\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.971837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-dshm\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.971837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971719    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.971837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.971837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971810    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.971837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.972061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-home\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.972061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971914    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.972061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971948    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.972061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.971991    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsm4g\" (UniqueName: \"kubernetes.io/projected/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kube-api-access-jsm4g\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.972061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.972022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:17.972290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.972115    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-model-cache\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.972290 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.972188    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.973874 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.973855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-dshm\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.974031 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.974015    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:17.982159 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:17.982134    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsm4g\" (UniqueName: \"kubernetes.io/projected/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kube-api-access-jsm4g\") pod \"custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:18.073218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431aaef4-6c9c-420b-9ddf-3e170d653e56-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073518 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073290    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073518 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg642\" (UniqueName: \"kubernetes.io/projected/431aaef4-6c9c-420b-9ddf-3e170d653e56-kube-api-access-mg642\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073518 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073376    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073518 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073518 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073472    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073690    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073757    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073872 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073788    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.073929 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.073911    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.075659 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.075641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.075947 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.075928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431aaef4-6c9c-420b-9ddf-3e170d653e56-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.081639 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.081622    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:18.083169 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.083146    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg642\" (UniqueName: \"kubernetes.io/projected/431aaef4-6c9c-420b-9ddf-3e170d653e56-kube-api-access-mg642\") pod \"custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.105357 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.105329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:18.235480 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.235450    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"]
Jul 16 18:16:18.236973 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:16:18.236940    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f937ea_89e6_40c9_9ab3_b28aa9f9cde1.slice/crio-eac3dc4dcaefd9af1ea7a5a7d28e805a904002776ed100daffcef1b559295b45 WatchSource:0}: Error finding container eac3dc4dcaefd9af1ea7a5a7d28e805a904002776ed100daffcef1b559295b45: Status 404 returned error can't find the container with id eac3dc4dcaefd9af1ea7a5a7d28e805a904002776ed100daffcef1b559295b45
Jul 16 18:16:18.267392 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.267365    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"]
Jul 16 18:16:18.269446 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:16:18.269413    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod431aaef4_6c9c_420b_9ddf_3e170d653e56.slice/crio-5d45f7b9dc6a3d3cd1cc949d16104ed70b0baf884bf5a0540506a95b1b7daa6e WatchSource:0}: Error finding container 5d45f7b9dc6a3d3cd1cc949d16104ed70b0baf884bf5a0540506a95b1b7daa6e: Status 404 returned error can't find the container with id 5d45f7b9dc6a3d3cd1cc949d16104ed70b0baf884bf5a0540506a95b1b7daa6e
Jul 16 18:16:18.331654 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.331549    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerStarted","Data":"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"}
Jul 16 18:16:18.331654 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.331620    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:18.331654 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.331646    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerStarted","Data":"eac3dc4dcaefd9af1ea7a5a7d28e805a904002776ed100daffcef1b559295b45"}
Jul 16 18:16:18.333643 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.333506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" event={"ID":"431aaef4-6c9c-420b-9ddf-3e170d653e56","Type":"ContainerStarted","Data":"0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d"}
Jul 16 18:16:18.333643 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:18.333566    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" event={"ID":"431aaef4-6c9c-420b-9ddf-3e170d653e56","Type":"ContainerStarted","Data":"5d45f7b9dc6a3d3cd1cc949d16104ed70b0baf884bf5a0540506a95b1b7daa6e"}
Jul 16 18:16:19.344571 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:19.344245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerStarted","Data":"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"}
Jul 16 18:16:22.369860 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:22.369816    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerID="3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb" exitCode=0
Jul 16 18:16:22.370404 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:22.369894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerDied","Data":"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"}
Jul 16 18:16:22.371570 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:22.371547    2579 generic.go:356] "Generic (PLEG): container finished" podID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerID="0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d" exitCode=0
Jul 16 18:16:22.371699 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:22.371634    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" event={"ID":"431aaef4-6c9c-420b-9ddf-3e170d653e56","Type":"ContainerDied","Data":"0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d"}
Jul 16 18:16:23.384184 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:23.384139    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" event={"ID":"431aaef4-6c9c-420b-9ddf-3e170d653e56","Type":"ContainerStarted","Data":"ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c"}
Jul 16 18:16:23.386425 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:23.386402    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerStarted","Data":"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"}
Jul 16 18:16:23.413750 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:23.413688    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podStartSLOduration=2.836363144 podStartE2EDuration="6.413667771s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.577304627s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:16:23.410396887 +0000 UTC m=+2504.793866820" watchObservedRunningTime="2026-07-16 18:16:23.413667771 +0000 UTC m=+2504.797137705"
Jul 16 18:16:23.442202 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:23.442139    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podStartSLOduration=3.038047904 podStartE2EDuration="6.442124881s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.404076977s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:16:23.440043633 +0000 UTC m=+2504.823513568" watchObservedRunningTime="2026-07-16 18:16:23.442124881 +0000 UTC m=+2504.825594869"
Jul 16 18:16:28.082057 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.082008    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:28.082565 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.082093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:28.083642 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.083601    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:16:28.095081 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.095049    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:16:28.105632 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.105606    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:28.105781 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.105659    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:16:28.107613 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.107580    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:16:28.400072 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.399985    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"]
Jul 16 18:16:28.406171 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.406147    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.409106 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.409082    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 16 18:16:28.418450 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.418428    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"]
Jul 16 18:16:28.473110 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6h6\" (UniqueName: \"kubernetes.io/projected/52ed41f6-b825-4244-a708-14bb951cf779-kube-api-access-8g6h6\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.473110 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-home\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.473410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-dshm\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.473410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473159    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-model-cache\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.473410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473253    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-kserve-provision-location\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.473410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-tmp-dir\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.473410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.473402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed41f6-b825-4244-a708-14bb951cf779-tls-certs\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.574765 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.574733    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-home\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.574765 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.574765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-dshm\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575047 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.574782    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-model-cache\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575047 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.574826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-kserve-provision-location\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575277 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.574882    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-tmp-dir\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575400 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.575179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-home\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575400 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.575290    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-model-cache\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575400 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.575228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-kserve-provision-location\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575400 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.575391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed41f6-b825-4244-a708-14bb951cf779-tls-certs\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575630 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.575429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6h6\" (UniqueName: \"kubernetes.io/projected/52ed41f6-b825-4244-a708-14bb951cf779-kube-api-access-8g6h6\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.575630 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.575459    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-tmp-dir\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.577708 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.577683    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-dshm\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.577828 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.577813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed41f6-b825-4244-a708-14bb951cf779-tls-certs\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.585015 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.584989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6h6\" (UniqueName: \"kubernetes.io/projected/52ed41f6-b825-4244-a708-14bb951cf779-kube-api-access-8g6h6\") pod \"stop-feature-test-kserve-f98bfbff-nln7t\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.719497 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.719391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:28.890815 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:28.890776    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"]
Jul 16 18:16:28.892131 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:16:28.892097    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ed41f6_b825_4244_a708_14bb951cf779.slice/crio-80b5af7ff9400ecd8480b28d5fdf51d9d4242c139e3e72d5538936b31b819134 WatchSource:0}: Error finding container 80b5af7ff9400ecd8480b28d5fdf51d9d4242c139e3e72d5538936b31b819134: Status 404 returned error can't find the container with id 80b5af7ff9400ecd8480b28d5fdf51d9d4242c139e3e72d5538936b31b819134
Jul 16 18:16:29.428317 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:29.428254    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" event={"ID":"52ed41f6-b825-4244-a708-14bb951cf779","Type":"ContainerStarted","Data":"741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936"}
Jul 16 18:16:29.428812 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:29.428352    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" event={"ID":"52ed41f6-b825-4244-a708-14bb951cf779","Type":"ContainerStarted","Data":"80b5af7ff9400ecd8480b28d5fdf51d9d4242c139e3e72d5538936b31b819134"}
Jul 16 18:16:32.446135 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:32.446098    2579 generic.go:356] "Generic (PLEG): container finished" podID="52ed41f6-b825-4244-a708-14bb951cf779" containerID="741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936" exitCode=0
Jul 16 18:16:32.446591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:32.446176    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" event={"ID":"52ed41f6-b825-4244-a708-14bb951cf779","Type":"ContainerDied","Data":"741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936"}
Jul 16 18:16:33.456564 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:33.456526    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" event={"ID":"52ed41f6-b825-4244-a708-14bb951cf779","Type":"ContainerStarted","Data":"d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d"}
Jul 16 18:16:33.482090 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:33.482024    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podStartSLOduration=2.14283448 podStartE2EDuration="5.482005922s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.339171442s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:16:33.481244393 +0000 UTC m=+2514.864714344" watchObservedRunningTime="2026-07-16 18:16:33.482005922 +0000 UTC m=+2514.865475855"
Jul 16 18:16:33.720570 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:33.720462    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:38.082301 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:38.082147    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:16:38.106192 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:38.106150    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:16:38.719748 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:38.719704    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:16:38.721743 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:38.721706    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:16:48.083231 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:48.083178    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:16:48.106099 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:48.106045    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:16:48.720094 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:48.720035    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:16:58.082822 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:58.082764    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:16:58.106035 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:58.105984    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:16:58.720901 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:16:58.720851    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:17:08.082240 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:08.082188    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:17:08.106357 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:08.106306    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:17:08.720033 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:08.719973    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:17:18.082672 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:18.082601    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:17:18.106761 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:18.106717    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:17:18.720386 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:18.720334    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:17:28.083246 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:28.083101    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:17:28.105783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:28.105728    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:17:28.719848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:28.719795    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:17:38.083128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:38.083064    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:17:38.106207 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:38.106150    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:17:38.720490 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:38.720440    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:17:48.082920 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:48.082861    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:17:48.106344 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:48.106305    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:17:48.720030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:48.719984    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:17:58.082318 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:58.082251    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:17:58.106430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:58.106393    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:17:58.720819 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:17:58.720770    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:18:08.083099 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:08.082969    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:18:08.105783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:08.105746    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:18:08.720709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:08.720663    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:18:18.082603 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:18.082547    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:18:18.105934 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:18.105891    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:18:18.720712 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:18.720669    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:18:28.082753 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:28.082701    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:18:28.106617 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:28.106574    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:18:28.720537 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:28.720491    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:18:38.082446 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:38.082395    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:18:38.106191 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:38.106147    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:18:38.720483 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:38.720439    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:18:48.083103 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:48.083050    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:18:48.106698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:48.106651    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:18:48.720019 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:48.719973    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:18:58.082627 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:58.082576    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:18:58.106597 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:58.106552    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:18:58.720561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:18:58.720514    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:19:08.082455 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:08.082407    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:19:08.106514 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:08.106475    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:19:08.720356 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:08.720315    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:19:18.083101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:18.083039    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:19:18.106130 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:18.106090    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:19:18.720395 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:18.720348    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:19:28.082855 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:28.082809    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:19:28.106225 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:28.106176    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 16 18:19:28.720578 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:28.720533    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:19:38.082567 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:38.082436    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8001/health\": dial tcp 10.133.0.66:8001: connect: connection refused"
Jul 16 18:19:38.116256 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:38.116225    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:19:38.124905 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:38.124876    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:19:38.720527 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:38.720455    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:19:48.092171 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:48.092135    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:19:48.103959 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:48.103933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:19:48.372808 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:48.372710    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:19:48.378102 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:48.378069    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:19:48.729660 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:48.729582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:19:48.737390 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:48.737357    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:19:51.252017 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:51.251984    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"]
Jul 16 18:19:57.197404 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:57.197356    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"]
Jul 16 18:19:57.199859 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:19:57.199831    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"]
Jul 16 18:20:06.278939 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:06.278890    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" containerID="cri-o://d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d" gracePeriod=45
Jul 16 18:20:06.731174 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:06.731128    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/health\": dial tcp 10.133.0.68:8000: connect: connection refused"
Jul 16 18:20:07.037311 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.037281    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:20:07.210186 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210147    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-model-cache\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210208    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed41f6-b825-4244-a708-14bb951cf779-tls-certs\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210246    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-kserve-provision-location\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210337    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-dshm\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210370    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8g6h6\" (UniqueName: \"kubernetes.io/projected/52ed41f6-b825-4244-a708-14bb951cf779-kube-api-access-8g6h6\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210387    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-home\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210403 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210403    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-tmp-dir\") pod \"52ed41f6-b825-4244-a708-14bb951cf779\" (UID: \"52ed41f6-b825-4244-a708-14bb951cf779\") "
Jul 16 18:20:07.210735 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-model-cache" (OuterVolumeSpecName: "model-cache") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:07.210735 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.210719    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.211191 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.211156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-home" (OuterVolumeSpecName: "home") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:07.212679 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.212648    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ed41f6-b825-4244-a708-14bb951cf779-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:20:07.212949 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.212925    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-dshm" (OuterVolumeSpecName: "dshm") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:07.213051 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.212961    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ed41f6-b825-4244-a708-14bb951cf779-kube-api-access-8g6h6" (OuterVolumeSpecName: "kube-api-access-8g6h6") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "kube-api-access-8g6h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:07.222888 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.222854    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:07.278364 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.278276    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "52ed41f6-b825-4244-a708-14bb951cf779" (UID: "52ed41f6-b825-4244-a708-14bb951cf779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:07.312098 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.312062    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.312098 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.312096    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g6h6\" (UniqueName: \"kubernetes.io/projected/52ed41f6-b825-4244-a708-14bb951cf779-kube-api-access-8g6h6\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.312098 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.312107    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.312585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.312122    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.312585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.312132    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ed41f6-b825-4244-a708-14bb951cf779-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.312585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.312145    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/52ed41f6-b825-4244-a708-14bb951cf779-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:07.829698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.829664    2579 generic.go:356] "Generic (PLEG): container finished" podID="52ed41f6-b825-4244-a708-14bb951cf779" containerID="d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d" exitCode=0
Jul 16 18:20:07.829877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.829717    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"
Jul 16 18:20:07.829877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.829754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" event={"ID":"52ed41f6-b825-4244-a708-14bb951cf779","Type":"ContainerDied","Data":"d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d"}
Jul 16 18:20:07.829877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.829814    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t" event={"ID":"52ed41f6-b825-4244-a708-14bb951cf779","Type":"ContainerDied","Data":"80b5af7ff9400ecd8480b28d5fdf51d9d4242c139e3e72d5538936b31b819134"}
Jul 16 18:20:07.829877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.829837    2579 scope.go:117] "RemoveContainer" containerID="d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d"
Jul 16 18:20:07.847318 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.847296    2579 scope.go:117] "RemoveContainer" containerID="741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936"
Jul 16 18:20:07.856349 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.856323    2579 scope.go:117] "RemoveContainer" containerID="d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d"
Jul 16 18:20:07.856634 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:07.856615    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d\": container with ID starting with d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d not found: ID does not exist" containerID="d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d"
Jul 16 18:20:07.856674 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.856646    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d"} err="failed to get container status \"d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d\": rpc error: code = NotFound desc = could not find container \"d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d\": container with ID starting with d3baceb6d7e7d8dd6d8ffc0c688f44de9789d38e6dc21f31b083dfcd08f3349d not found: ID does not exist"
Jul 16 18:20:07.856674 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.856664    2579 scope.go:117] "RemoveContainer" containerID="741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936"
Jul 16 18:20:07.856939 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:07.856897    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936\": container with ID starting with 741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936 not found: ID does not exist" containerID="741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936"
Jul 16 18:20:07.856976 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.856946    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936"} err="failed to get container status \"741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936\": rpc error: code = NotFound desc = could not find container \"741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936\": container with ID starting with 741938712c50493a9092560455bee1b754d21a54fc3c07b05d83e60202cdd936 not found: ID does not exist"
Jul 16 18:20:07.863992 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.863967    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"]
Jul 16 18:20:07.868953 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:07.868930    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-f98bfbff-nln7t"]
Jul 16 18:20:09.103372 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:09.103341    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52ed41f6-b825-4244-a708-14bb951cf779" path="/var/lib/kubelet/pods/52ed41f6-b825-4244-a708-14bb951cf779/volumes"
Jul 16 18:20:12.221319 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:12.221261    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main" containerID="cri-o://ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c" gracePeriod=45
Jul 16 18:20:12.228727 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:12.228678    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main" containerID="cri-o://ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6" gracePeriod=45
Jul 16 18:20:13.046520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.046485    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="llm-d-routing-sidecar" containerID="cri-o://f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a" gracePeriod=45
Jul 16 18:20:13.184363 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.184337    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:20:13.298409 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.298382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:20:13.369128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369096    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-kserve-provision-location\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369128 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431aaef4-6c9c-420b-9ddf-3e170d653e56-tls-certs\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369414 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369163    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-home\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369414 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-model-cache\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369414 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369218    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg642\" (UniqueName: \"kubernetes.io/projected/431aaef4-6c9c-420b-9ddf-3e170d653e56-kube-api-access-mg642\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369414 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369236    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-tmp-dir\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369414 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369297    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-dshm\") pod \"431aaef4-6c9c-420b-9ddf-3e170d653e56\" (UID: \"431aaef4-6c9c-420b-9ddf-3e170d653e56\") "
Jul 16 18:20:13.369682 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369470    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-model-cache" (OuterVolumeSpecName: "model-cache") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.369682 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369659    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.369911 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.369885    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-home" (OuterVolumeSpecName: "home") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.371538 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.371494    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/431aaef4-6c9c-420b-9ddf-3e170d653e56-kube-api-access-mg642" (OuterVolumeSpecName: "kube-api-access-mg642") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "kube-api-access-mg642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:13.371676 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.371650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/431aaef4-6c9c-420b-9ddf-3e170d653e56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:20:13.372179 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.372128    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-dshm" (OuterVolumeSpecName: "dshm") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.383931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.383883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.438296 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.438238    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "431aaef4-6c9c-420b-9ddf-3e170d653e56" (UID: "431aaef4-6c9c-420b-9ddf-3e170d653e56"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.470129 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470034    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kserve-provision-location\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470129 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470078    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-model-cache\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470129 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470103    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-home\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470129 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470126    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tmp-dir\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470192    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tls-certs\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470224    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jsm4g\" (UniqueName: \"kubernetes.io/projected/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kube-api-access-jsm4g\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470259    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-dshm\") pod \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\" (UID: \"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1\") "
Jul 16 18:20:13.470491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470454    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.470709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470623    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/431aaef4-6c9c-420b-9ddf-3e170d653e56-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.470709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470645    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.470709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470658    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.470709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470672    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg642\" (UniqueName: \"kubernetes.io/projected/431aaef4-6c9c-420b-9ddf-3e170d653e56-kube-api-access-mg642\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.470709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470687    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.470709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470700    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.471032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470714    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/431aaef4-6c9c-420b-9ddf-3e170d653e56-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.471032 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.470869    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-home" (OuterVolumeSpecName: "home") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.472752 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.472704    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:20:13.472969 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.472949    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-dshm" (OuterVolumeSpecName: "dshm") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.473066 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.473011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kube-api-access-jsm4g" (OuterVolumeSpecName: "kube-api-access-jsm4g") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "kube-api-access-jsm4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:20:13.486612 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.486566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.540402 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.540359    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" (UID: "f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:20:13.571541 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.571501    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.571541 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.571533    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.571756 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.571553    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.571756 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.571561    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.571756 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.571571    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsm4g\" (UniqueName: \"kubernetes.io/projected/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-kube-api-access-jsm4g\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.571756 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.571578    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:20:13.863894 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.863849    2579 generic.go:356] "Generic (PLEG): container finished" podID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerID="ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c" exitCode=0
Jul 16 18:20:13.864121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.863908    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"
Jul 16 18:20:13.864121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.863941    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" event={"ID":"431aaef4-6c9c-420b-9ddf-3e170d653e56","Type":"ContainerDied","Data":"ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c"}
Jul 16 18:20:13.864121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.864010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn" event={"ID":"431aaef4-6c9c-420b-9ddf-3e170d653e56","Type":"ContainerDied","Data":"5d45f7b9dc6a3d3cd1cc949d16104ed70b0baf884bf5a0540506a95b1b7daa6e"}
Jul 16 18:20:13.864121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.864042    2579 scope.go:117] "RemoveContainer" containerID="ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c"
Jul 16 18:20:13.866122 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.866097    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerID="ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6" exitCode=0
Jul 16 18:20:13.866122 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.866119    2579 generic.go:356] "Generic (PLEG): container finished" podID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerID="f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a" exitCode=0
Jul 16 18:20:13.866396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.866155    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerDied","Data":"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"}
Jul 16 18:20:13.866396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.866175    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"
Jul 16 18:20:13.866396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.866201    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerDied","Data":"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"}
Jul 16 18:20:13.866396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.866220    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6" event={"ID":"f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1","Type":"ContainerDied","Data":"eac3dc4dcaefd9af1ea7a5a7d28e805a904002776ed100daffcef1b559295b45"}
Jul 16 18:20:13.878245 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.877049    2579 scope.go:117] "RemoveContainer" containerID="0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d"
Jul 16 18:20:13.887630 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.887609    2579 scope.go:117] "RemoveContainer" containerID="ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c"
Jul 16 18:20:13.887884 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:13.887866    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c\": container with ID starting with ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c not found: ID does not exist" containerID="ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c"
Jul 16 18:20:13.887956 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.887891    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c"} err="failed to get container status \"ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c\": rpc error: code = NotFound desc = could not find container \"ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c\": container with ID starting with ac76c5c26274f4a40441b1b581aafb0a39182eee2d18f7c100beb2bd117c3b8c not found: ID does not exist"
Jul 16 18:20:13.887956 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.887906    2579 scope.go:117] "RemoveContainer" containerID="0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d"
Jul 16 18:20:13.888141 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:13.888122    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d\": container with ID starting with 0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d not found: ID does not exist" containerID="0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d"
Jul 16 18:20:13.888182 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.888149    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d"} err="failed to get container status \"0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d\": rpc error: code = NotFound desc = could not find container \"0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d\": container with ID starting with 0a67aebe11e9dcd5aea42982501a2fed10597267571f62ca30f6262d656afb0d not found: ID does not exist"
Jul 16 18:20:13.888182 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.888165    2579 scope.go:117] "RemoveContainer" containerID="ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"
Jul 16 18:20:13.896362 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.896338    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"]
Jul 16 18:20:13.897724 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.897674    2579 scope.go:117] "RemoveContainer" containerID="3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"
Jul 16 18:20:13.900195 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.900171    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-589f5b6bd7-9mmxn"]
Jul 16 18:20:13.906896 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.906862    2579 scope.go:117] "RemoveContainer" containerID="f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"
Jul 16 18:20:13.915489 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.915463    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"]
Jul 16 18:20:13.915728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.915711    2579 scope.go:117] "RemoveContainer" containerID="ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"
Jul 16 18:20:13.916032 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:13.916013    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6\": container with ID starting with ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6 not found: ID does not exist" containerID="ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"
Jul 16 18:20:13.916105 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916046    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"} err="failed to get container status \"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6\": rpc error: code = NotFound desc = could not find container \"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6\": container with ID starting with ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6 not found: ID does not exist"
Jul 16 18:20:13.916105 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916072    2579 scope.go:117] "RemoveContainer" containerID="3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"
Jul 16 18:20:13.916327 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:13.916305    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb\": container with ID starting with 3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb not found: ID does not exist" containerID="3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"
Jul 16 18:20:13.916367 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916333    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"} err="failed to get container status \"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb\": rpc error: code = NotFound desc = could not find container \"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb\": container with ID starting with 3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb not found: ID does not exist"
Jul 16 18:20:13.916367 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916349    2579 scope.go:117] "RemoveContainer" containerID="f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"
Jul 16 18:20:13.916581 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:20:13.916564    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a\": container with ID starting with f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a not found: ID does not exist" containerID="f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"
Jul 16 18:20:13.916651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916589    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"} err="failed to get container status \"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a\": rpc error: code = NotFound desc = could not find container \"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a\": container with ID starting with f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a not found: ID does not exist"
Jul 16 18:20:13.916651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916608    2579 scope.go:117] "RemoveContainer" containerID="ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"
Jul 16 18:20:13.916837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916820    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6"} err="failed to get container status \"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6\": rpc error: code = NotFound desc = could not find container \"ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6\": container with ID starting with ce0af4cdc17114e69c9fe022a92d68ca78daa3f1a5e79c49462cf207f5de1bd6 not found: ID does not exist"
Jul 16 18:20:13.916885 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.916838    2579 scope.go:117] "RemoveContainer" containerID="3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"
Jul 16 18:20:13.917075 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.917054    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb"} err="failed to get container status \"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb\": rpc error: code = NotFound desc = could not find container \"3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb\": container with ID starting with 3717c5e2131e7c88d37424ef12f39a3df62df0bee9e701da849591c6a1eb49cb not found: ID does not exist"
Jul 16 18:20:13.917165 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.917076    2579 scope.go:117] "RemoveContainer" containerID="f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"
Jul 16 18:20:13.917355 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.917298    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a"} err="failed to get container status \"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a\": rpc error: code = NotFound desc = could not find container \"f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a\": container with ID starting with f4030405fea9dc4f7510ef29cdb35a7dd54e6cdec398850a490e4c373bdf010a not found: ID does not exist"
Jul 16 18:20:13.920607 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:13.920585    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-5d9b9b4785-cb2v6"]
Jul 16 18:20:14.620709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.620671    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"]
Jul 16 18:20:14.621092 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621051    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="storage-initializer"
Jul 16 18:20:14.621092 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621063    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="storage-initializer"
Jul 16 18:20:14.621092 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621071    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main"
Jul 16 18:20:14.621092 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621076    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main"
Jul 16 18:20:14.621092 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621089    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="llm-d-routing-sidecar"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621095    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="llm-d-routing-sidecar"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621103    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="storage-initializer"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621112    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="storage-initializer"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621121    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621128    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621135    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621141    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621150    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="storage-initializer"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621154    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="storage-initializer"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621216    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="52ed41f6-b825-4244-a708-14bb951cf779" containerName="main"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621229    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" containerName="main"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621236    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="llm-d-routing-sidecar"
Jul 16 18:20:14.621309 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.621242    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" containerName="main"
Jul 16 18:20:14.626890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.626865    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.631889 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.631868    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 16 18:20:14.632340 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.632318    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:20:14.632562 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.632547    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 18:20:14.632639 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.632610    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:20:14.632745 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.632732    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:20:14.633154 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.633137    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:20:14.641279 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.641237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"]
Jul 16 18:20:14.781808 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.781770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-home\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.781808 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.781808    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tls-certs\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.782020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.781835    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz9n8\" (UniqueName: \"kubernetes.io/projected/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kube-api-access-mz9n8\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.782020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.781922    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-dshm\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.782020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.781955    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce68858-ed45-4208-98d5-bdcadc1dcaa3-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.782166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.782030    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tmp-dir\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.782166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.782064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-model-cache\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.782166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.782103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883464    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-home\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tls-certs\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz9n8\" (UniqueName: \"kubernetes.io/projected/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kube-api-access-mz9n8\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883910 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883591    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-dshm\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883910 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883618    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce68858-ed45-4208-98d5-bdcadc1dcaa3-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.883910 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883833    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tmp-dir\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.884071 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.883954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.884071 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.884005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-home\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.884281 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.884234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tmp-dir\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.884415 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.884312    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce68858-ed45-4208-98d5-bdcadc1dcaa3-cabundle-cert\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.884415 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.884361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-model-cache\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.884594 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.884576    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-model-cache\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.886497 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.886464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-dshm\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.886659 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.886641    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tls-certs\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.896483 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.896461    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz9n8\" (UniqueName: \"kubernetes.io/projected/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kube-api-access-mz9n8\") pod \"lora-single-adapter-test-kserve-6c7646bb6b-7wx5b\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:14.937555 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:14.937512    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:15.103433 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:20:15.103402    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce68858_ed45_4208_98d5_bdcadc1dcaa3.slice/crio-c343da155ab999d6afe370a275195558f60eeed14d03e618b7640306c61acbe1 WatchSource:0}: Error finding container c343da155ab999d6afe370a275195558f60eeed14d03e618b7640306c61acbe1: Status 404 returned error can't find the container with id c343da155ab999d6afe370a275195558f60eeed14d03e618b7640306c61acbe1
Jul 16 18:20:15.104132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:15.104108    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="431aaef4-6c9c-420b-9ddf-3e170d653e56" path="/var/lib/kubelet/pods/431aaef4-6c9c-420b-9ddf-3e170d653e56/volumes"
Jul 16 18:20:15.104720 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:15.104700    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1" path="/var/lib/kubelet/pods/f2f937ea-89e6-40c9-9ab3-b28aa9f9cde1/volumes"
Jul 16 18:20:15.105096 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:15.105083    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"]
Jul 16 18:20:15.880550 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:15.880517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" event={"ID":"fce68858-ed45-4208-98d5-bdcadc1dcaa3","Type":"ContainerStarted","Data":"4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf"}
Jul 16 18:20:15.881015 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:15.880575    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" event={"ID":"fce68858-ed45-4208-98d5-bdcadc1dcaa3","Type":"ContainerStarted","Data":"c343da155ab999d6afe370a275195558f60eeed14d03e618b7640306c61acbe1"}
Jul 16 18:20:18.900988 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:18.900950    2579 generic.go:356] "Generic (PLEG): container finished" podID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerID="4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf" exitCode=0
Jul 16 18:20:18.901503 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:18.901011    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" event={"ID":"fce68858-ed45-4208-98d5-bdcadc1dcaa3","Type":"ContainerDied","Data":"4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf"}
Jul 16 18:20:19.909124 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:19.909094    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" event={"ID":"fce68858-ed45-4208-98d5-bdcadc1dcaa3","Type":"ContainerStarted","Data":"2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26"}
Jul 16 18:20:19.937805 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:19.937777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:19.943693 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:19.943638    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podStartSLOduration=2.314297099 podStartE2EDuration="5.94362111s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.629324011s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:20:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:20:19.941826479 +0000 UTC m=+2741.325296413" watchObservedRunningTime="2026-07-16 18:20:19.94362111 +0000 UTC m=+2741.327091044"
Jul 16 18:20:24.938399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:24.938366    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:20:24.940297 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:24.940257    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:20:34.938452 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:34.938404    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:20:40.869053 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.869008    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"]
Jul 16 18:20:40.872253 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.872232    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.875112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.875086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-kh6mm\""
Jul 16 18:20:40.875298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.875166    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 16 18:20:40.892255 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.892222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"]
Jul 16 18:20:40.905756 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.903634    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"]
Jul 16 18:20:40.908605 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.908119    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:40.928588 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.928552    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"]
Jul 16 18:20:40.933533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933495    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-home\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.933705 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.933705 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-model-cache\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.933705 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933627    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fb203-2156-4ba6-82be-346b9852c5c8-tls-certs\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.933705 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.933705 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ms7g\" (UniqueName: \"kubernetes.io/projected/358fb203-2156-4ba6-82be-346b9852c5c8-kube-api-access-7ms7g\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:40.933953 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:40.933706    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-dshm\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034376 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034339    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034386    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd9jm\" (UniqueName: \"kubernetes.io/projected/6fd84871-6341-436c-9fb0-edefbed5f4e6-kube-api-access-hd9jm\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034455    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-home\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034525    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034541    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034728    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-home\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034759    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-model-cache\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fb203-2156-4ba6-82be-346b9852c5c8-tls-certs\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034845    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ms7g\" (UniqueName: \"kubernetes.io/projected/358fb203-2156-4ba6-82be-346b9852c5c8-kube-api-access-7ms7g\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034876    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd84871-6341-436c-9fb0-edefbed5f4e6-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.034945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034902    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-home\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.034945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.034918    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-dshm\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.035254 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.035039    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-tmp-dir\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.035254 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.035123    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-model-cache\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.035465 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.035443    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.037737 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.037714    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-dshm\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.038029 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.038010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fb203-2156-4ba6-82be-346b9852c5c8-tls-certs\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.043786 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.043764    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ms7g\" (UniqueName: \"kubernetes.io/projected/358fb203-2156-4ba6-82be-346b9852c5c8-kube-api-access-7ms7g\") pod \"router-with-refs-pd-test-kserve-b4685cc89-zfv4b\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.136474 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136372    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.136474 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd9jm\" (UniqueName: \"kubernetes.io/projected/6fd84871-6341-436c-9fb0-edefbed5f4e6-kube-api-access-hd9jm\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.136722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.136722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136504    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.136722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136532    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.136722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-home\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.136722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd84871-6341-436c-9fb0-edefbed5f4e6-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.137061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.136896    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.137061 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.137021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.137156 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.137092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.137191 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.137150    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-home\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.138831 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.138796    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.139614 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.139591    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd84871-6341-436c-9fb0-edefbed5f4e6-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.145896 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.145866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd9jm\" (UniqueName: \"kubernetes.io/projected/6fd84871-6341-436c-9fb0-edefbed5f4e6-kube-api-access-hd9jm\") pod \"router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.185036 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.184991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:41.230467 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.230403    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:41.367161 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.367114    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"]
Jul 16 18:20:41.371235 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:20:41.371201    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358fb203_2156_4ba6_82be_346b9852c5c8.slice/crio-4d22f76524aada8ed702881af8133f3a9bf4e5cf875635ea7507a03aa0ebbbb9 WatchSource:0}: Error finding container 4d22f76524aada8ed702881af8133f3a9bf4e5cf875635ea7507a03aa0ebbbb9: Status 404 returned error can't find the container with id 4d22f76524aada8ed702881af8133f3a9bf4e5cf875635ea7507a03aa0ebbbb9
Jul 16 18:20:41.401586 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:41.401561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"]
Jul 16 18:20:41.414580 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:20:41.414552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd84871_6341_436c_9fb0_edefbed5f4e6.slice/crio-f22d6e38428933de98895e32c7d6835c5371de3d4c296b1c836892f440bc1106 WatchSource:0}: Error finding container f22d6e38428933de98895e32c7d6835c5371de3d4c296b1c836892f440bc1106: Status 404 returned error can't find the container with id f22d6e38428933de98895e32c7d6835c5371de3d4c296b1c836892f440bc1106
Jul 16 18:20:42.044645 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:42.044596    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" event={"ID":"6fd84871-6341-436c-9fb0-edefbed5f4e6","Type":"ContainerStarted","Data":"86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723"}
Jul 16 18:20:42.045112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:42.044709    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" event={"ID":"6fd84871-6341-436c-9fb0-edefbed5f4e6","Type":"ContainerStarted","Data":"f22d6e38428933de98895e32c7d6835c5371de3d4c296b1c836892f440bc1106"}
Jul 16 18:20:42.046556 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:42.046525    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerStarted","Data":"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"}
Jul 16 18:20:42.046692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:42.046581    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerStarted","Data":"4d22f76524aada8ed702881af8133f3a9bf4e5cf875635ea7507a03aa0ebbbb9"}
Jul 16 18:20:42.046692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:42.046636    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:43.057413 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:43.057370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerStarted","Data":"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"}
Jul 16 18:20:44.938763 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:44.938705    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:20:45.081833 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:45.081795    2579 generic.go:356] "Generic (PLEG): container finished" podID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerID="86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723" exitCode=0
Jul 16 18:20:45.082033 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:45.081976    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" event={"ID":"6fd84871-6341-436c-9fb0-edefbed5f4e6","Type":"ContainerDied","Data":"86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723"}
Jul 16 18:20:46.093473 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:46.093431    2579 generic.go:356] "Generic (PLEG): container finished" podID="358fb203-2156-4ba6-82be-346b9852c5c8" containerID="2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532" exitCode=0
Jul 16 18:20:46.093911 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:46.093497    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerDied","Data":"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"}
Jul 16 18:20:46.096160 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:46.096137    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" event={"ID":"6fd84871-6341-436c-9fb0-edefbed5f4e6","Type":"ContainerStarted","Data":"e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3"}
Jul 16 18:20:46.148848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:46.148804    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podStartSLOduration=2.637286579 podStartE2EDuration="6.148785915s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.511499336s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:20:46.144458431 +0000 UTC m=+2767.527928366" watchObservedRunningTime="2026-07-16 18:20:46.148785915 +0000 UTC m=+2767.532255853"
Jul 16 18:20:47.120495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:47.120446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerStarted","Data":"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"}
Jul 16 18:20:47.163365 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:47.163253    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podStartSLOduration=3.713679254 podStartE2EDuration="7.163229836s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.449550582s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:20:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:20:47.156362413 +0000 UTC m=+2768.539832349" watchObservedRunningTime="2026-07-16 18:20:47.163229836 +0000 UTC m=+2768.546699773"
Jul 16 18:20:51.185601 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.185558    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:51.186418 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.185622    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:51.187281 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.187235    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:20:51.199671 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.199643    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:20:51.230834 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.230701    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:51.231095 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.231078    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:20:51.237184 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:51.237135    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:20:54.938829 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:20:54.938776    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:21:01.186600 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:01.186491    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:21:01.231134 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:01.231086    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:21:04.938668 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:04.938612    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:21:11.185949 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:11.185903    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:21:11.231496 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:11.231450    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:21:14.938815 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:14.938766    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:21:21.185872 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:21.185825    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:21:21.231664 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:21.231620    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:21:24.938586 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:24.938526    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:21:31.185743 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:31.185675    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:21:31.231091 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:31.231034    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:21:34.938218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:34.938150    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:21:41.185960 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:41.185898    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:21:41.231462 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:41.231408    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:21:44.955692 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:44.955643    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:21:44.964813 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:44.964782    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:21:51.185866 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:51.185811    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:21:51.231210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:21:51.231148    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:01.186016 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:01.185967    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:22:01.231677 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:01.231620    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:11.185615 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:11.185572    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:22:11.231615 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:11.231573    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:21.185893 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:21.185846    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:22:21.230819 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:21.230778    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:31.186339 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:31.186217    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:22:31.231262 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:31.231214    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:41.185505 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:41.185447    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:22:41.230945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:41.230904    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:42.486579 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:42.486527    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"]
Jul 16 18:22:51.185722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:51.185671    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:22:51.231703 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:51.231660    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:22:57.521994 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:57.521909    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" containerID="cri-o://2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26" gracePeriod=45
Jul 16 18:22:57.957443 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:57.957395    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 16 18:22:58.796945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.796916    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:22:58.942796 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.942694    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mz9n8\" (UniqueName: \"kubernetes.io/projected/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kube-api-access-mz9n8\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.942796 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.942758    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kserve-provision-location\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.942796 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.942790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-dshm\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.943101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.942865    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-model-cache\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.943101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.942895    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tmp-dir\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.943101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.942979    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce68858-ed45-4208-98d5-bdcadc1dcaa3-cabundle-cert\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.943101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.943016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tls-certs\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.943101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.943069    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-home\") pod \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\" (UID: \"fce68858-ed45-4208-98d5-bdcadc1dcaa3\") "
Jul 16 18:22:58.943408 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.943098    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-model-cache" (OuterVolumeSpecName: "model-cache") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:22:58.943408 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.943393    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce68858-ed45-4208-98d5-bdcadc1dcaa3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:22:58.943520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.943441    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:58.943797 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.943769    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-home" (OuterVolumeSpecName: "home") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:22:58.945341 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.945308    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kube-api-access-mz9n8" (OuterVolumeSpecName: "kube-api-access-mz9n8") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "kube-api-access-mz9n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:22:58.945483 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.945458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-dshm" (OuterVolumeSpecName: "dshm") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:22:58.945634 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.945609    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:22:58.954068 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:58.954041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:22:59.012249 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.012200    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fce68858-ed45-4208-98d5-bdcadc1dcaa3" (UID: "fce68858-ed45-4208-98d5-bdcadc1dcaa3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:22:59.044553 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044521    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.044553 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044551    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fce68858-ed45-4208-98d5-bdcadc1dcaa3-cabundle-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.044703 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044563    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fce68858-ed45-4208-98d5-bdcadc1dcaa3-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.044703 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044570    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.044703 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044579    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz9n8\" (UniqueName: \"kubernetes.io/projected/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kube-api-access-mz9n8\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.044703 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044588    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.044703 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.044610    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fce68858-ed45-4208-98d5-bdcadc1dcaa3-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:22:59.067327 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.067293    2579 generic.go:356] "Generic (PLEG): container finished" podID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerID="2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26" exitCode=0
Jul 16 18:22:59.067508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.067378    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"
Jul 16 18:22:59.067508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.067418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" event={"ID":"fce68858-ed45-4208-98d5-bdcadc1dcaa3","Type":"ContainerDied","Data":"2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26"}
Jul 16 18:22:59.067508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.067471    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b" event={"ID":"fce68858-ed45-4208-98d5-bdcadc1dcaa3","Type":"ContainerDied","Data":"c343da155ab999d6afe370a275195558f60eeed14d03e618b7640306c61acbe1"}
Jul 16 18:22:59.067508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.067493    2579 scope.go:117] "RemoveContainer" containerID="2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26"
Jul 16 18:22:59.079026 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.079010    2579 scope.go:117] "RemoveContainer" containerID="4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf"
Jul 16 18:22:59.087624 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.087608    2579 scope.go:117] "RemoveContainer" containerID="2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26"
Jul 16 18:22:59.087871 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:22:59.087853    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26\": container with ID starting with 2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26 not found: ID does not exist" containerID="2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26"
Jul 16 18:22:59.087947 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.087879    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26"} err="failed to get container status \"2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26\": rpc error: code = NotFound desc = could not find container \"2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26\": container with ID starting with 2de3cd7e2582ca576574a0e40093fb29efed401716849b76ee8c6eafdd4c9b26 not found: ID does not exist"
Jul 16 18:22:59.087947 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.087897    2579 scope.go:117] "RemoveContainer" containerID="4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf"
Jul 16 18:22:59.088141 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:22:59.088127    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf\": container with ID starting with 4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf not found: ID does not exist" containerID="4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf"
Jul 16 18:22:59.088183 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.088144    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf"} err="failed to get container status \"4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf\": rpc error: code = NotFound desc = could not find container \"4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf\": container with ID starting with 4b02312b097c43c49a39adaa0754de157967eddbacf6eaad9c161949bb4b58cf not found: ID does not exist"
Jul 16 18:22:59.093672 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.093643    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"]
Jul 16 18:22:59.103556 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:22:59.103533    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-6c7646bb6b-7wx5b"]
Jul 16 18:23:01.103563 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:01.103525    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" path="/var/lib/kubelet/pods/fce68858-ed45-4208-98d5-bdcadc1dcaa3/volumes"
Jul 16 18:23:01.185678 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:01.185630    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:23:01.231859 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:01.231817    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:23:11.186430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:11.186386    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:23:11.231334 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:11.231290    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:23:14.082820 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.082788    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"]
Jul 16 18:23:14.083302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.083189    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="storage-initializer"
Jul 16 18:23:14.083302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.083202    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="storage-initializer"
Jul 16 18:23:14.083302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.083222    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main"
Jul 16 18:23:14.083302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.083228    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main"
Jul 16 18:23:14.083469 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.083328    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce68858-ed45-4208-98d5-bdcadc1dcaa3" containerName="main"
Jul 16 18:23:14.088349 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.088330    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.092018 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.091997    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 18:23:14.092137 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.092115    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 16 18:23:14.098248 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.098227    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"]
Jul 16 18:23:14.190425 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190375    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190498    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-home\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190594    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksctz\" (UniqueName: \"kubernetes.io/projected/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kube-api-access-ksctz\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190634    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tls-certs\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-model-cache\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.190870 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.190815    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-dshm\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292289 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-home\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292331    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksctz\" (UniqueName: \"kubernetes.io/projected/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kube-api-access-ksctz\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tls-certs\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292451    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-model-cache\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292732 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292732 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292579    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-dshm\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292732 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292894 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292732    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-home\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292894 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-model-cache\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.292969 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.292916    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.293213 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.293184    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.293352 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.293283    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.294840 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.294817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-dshm\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.295160 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.295140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tls-certs\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.302424 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.302396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksctz\" (UniqueName: \"kubernetes.io/projected/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kube-api-access-ksctz\") pod \"lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.399581 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.399490    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:14.552787 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:14.552756    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"]
Jul 16 18:23:14.553497 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:23:14.553467    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode288b40e_5ffa_4fbe_9c1b_e98d835729a0.slice/crio-ab7aa86a48c52125833b4e6c66a30eb2e6874fa35d262f8288548e82be872592 WatchSource:0}: Error finding container ab7aa86a48c52125833b4e6c66a30eb2e6874fa35d262f8288548e82be872592: Status 404 returned error can't find the container with id ab7aa86a48c52125833b4e6c66a30eb2e6874fa35d262f8288548e82be872592
Jul 16 18:23:15.162798 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:15.162755    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" event={"ID":"e288b40e-5ffa-4fbe-9c1b-e98d835729a0","Type":"ContainerStarted","Data":"e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96"}
Jul 16 18:23:15.163212 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:15.162816    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" event={"ID":"e288b40e-5ffa-4fbe-9c1b-e98d835729a0","Type":"ContainerStarted","Data":"ab7aa86a48c52125833b4e6c66a30eb2e6874fa35d262f8288548e82be872592"}
Jul 16 18:23:19.188790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:19.188686    2579 generic.go:356] "Generic (PLEG): container finished" podID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerID="e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96" exitCode=0
Jul 16 18:23:19.188790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:19.188766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" event={"ID":"e288b40e-5ffa-4fbe-9c1b-e98d835729a0","Type":"ContainerDied","Data":"e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96"}
Jul 16 18:23:20.200775 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:20.200725    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" event={"ID":"e288b40e-5ffa-4fbe-9c1b-e98d835729a0","Type":"ContainerStarted","Data":"df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed"}
Jul 16 18:23:20.226910 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:20.226839    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podStartSLOduration=1.9899921950000001 podStartE2EDuration="6.226801489s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.236809294s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:23:20.224610311 +0000 UTC m=+2921.608080269" watchObservedRunningTime="2026-07-16 18:23:20.226801489 +0000 UTC m=+2921.610271424"
Jul 16 18:23:20.400186 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:20.400140    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:21.185592 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:21.185533    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:23:21.231744 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:21.231690    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:23:24.399903 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:24.399852    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:23:24.401463 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:24.401430    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:23:31.186566 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:31.186515    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:23:31.231004 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:31.230953    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:23:34.400132 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:34.400085    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:23:41.185644 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:41.185593    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8001/health\": dial tcp 10.133.0.70:8001: connect: connection refused"
Jul 16 18:23:41.231780 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:41.231735    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": dial tcp 10.133.0.71:8000: connect: connection refused"
Jul 16 18:23:44.400561 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:44.400519    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:23:51.195941 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:51.195906    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:23:51.216933 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:51.216905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:23:51.241197 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:51.241167    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:23:51.251166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:51.251136    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:23:54.400873 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:23:54.400825    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:24:01.820206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:01.820112    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"]
Jul 16 18:24:01.824105 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:01.824052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"]
Jul 16 18:24:04.400017 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:04.399974    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:24:14.400867 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:14.400815    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:24:16.847033 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:16.846955    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main" containerID="cri-o://e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3" gracePeriod=45
Jul 16 18:24:16.849764 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:16.849727    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main" containerID="cri-o://991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f" gracePeriod=45
Jul 16 18:24:17.648768 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:24:17.648737    2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358fb203_2156_4ba6_82be_346b9852c5c8.slice/crio-conmon-991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 18:24:17.663856 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:17.663825    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="llm-d-routing-sidecar" containerID="cri-o://175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747" gracePeriod=45
Jul 16 18:24:17.868117 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:17.868089    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:24:17.952314 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:17.952289    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:24:18.036910 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.036859    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fb203-2156-4ba6-82be-346b9852c5c8-tls-certs\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.036918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-dshm\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.036970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-dshm\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.037121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037006    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-model-cache\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037044    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hd9jm\" (UniqueName: \"kubernetes.io/projected/6fd84871-6341-436c-9fb0-edefbed5f4e6-kube-api-access-hd9jm\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.037121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037090    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-home\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.037121 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037119    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd84871-6341-436c-9fb0-edefbed5f4e6-tls-certs\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037148    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-tmp-dir\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037176    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-kserve-provision-location\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-kserve-provision-location\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037237    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-home\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-tmp-dir\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037319    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ms7g\" (UniqueName: \"kubernetes.io/projected/358fb203-2156-4ba6-82be-346b9852c5c8-kube-api-access-7ms7g\") pod \"358fb203-2156-4ba6-82be-346b9852c5c8\" (UID: \"358fb203-2156-4ba6-82be-346b9852c5c8\") "
Jul 16 18:24:18.037479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037428    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-model-cache\") pod \"6fd84871-6341-436c-9fb0-edefbed5f4e6\" (UID: \"6fd84871-6341-436c-9fb0-edefbed5f4e6\") "
Jul 16 18:24:18.038082 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.037940    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.039322 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.038810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-home" (OuterVolumeSpecName: "home") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.039322 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.038972    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-model-cache" (OuterVolumeSpecName: "model-cache") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.044175 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.044141    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-home" (OuterVolumeSpecName: "home") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.044308 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.044247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358fb203-2156-4ba6-82be-346b9852c5c8-kube-api-access-7ms7g" (OuterVolumeSpecName: "kube-api-access-7ms7g") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "kube-api-access-7ms7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:24:18.056693 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.056659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fd84871-6341-436c-9fb0-edefbed5f4e6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:24:18.056877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.056783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-dshm" (OuterVolumeSpecName: "dshm") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.056996 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.056906    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358fb203-2156-4ba6-82be-346b9852c5c8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:24:18.057836 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.057806    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-dshm" (OuterVolumeSpecName: "dshm") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.058691 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.058663    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd84871-6341-436c-9fb0-edefbed5f4e6-kube-api-access-hd9jm" (OuterVolumeSpecName: "kube-api-access-hd9jm") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "kube-api-access-hd9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:24:18.061137 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.061112    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.080945 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.080904    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.117906 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.117796    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fd84871-6341-436c-9fb0-edefbed5f4e6" (UID: "6fd84871-6341-436c-9fb0-edefbed5f4e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.128985 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.128944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "358fb203-2156-4ba6-82be-346b9852c5c8" (UID: "358fb203-2156-4ba6-82be-346b9852c5c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:24:18.138153 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138121    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138153 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138147    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6fd84871-6341-436c-9fb0-edefbed5f4e6-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138153 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138158    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138167    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138175    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138184    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138191    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138199    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ms7g\" (UniqueName: \"kubernetes.io/projected/358fb203-2156-4ba6-82be-346b9852c5c8-kube-api-access-7ms7g\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138206    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138214    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/358fb203-2156-4ba6-82be-346b9852c5c8-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138221    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138228    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6fd84871-6341-436c-9fb0-edefbed5f4e6-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138235    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/358fb203-2156-4ba6-82be-346b9852c5c8-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.138479 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.138243    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd9jm\" (UniqueName: \"kubernetes.io/projected/6fd84871-6341-436c-9fb0-edefbed5f4e6-kube-api-access-hd9jm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:24:18.649136 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649097    2579 generic.go:356] "Generic (PLEG): container finished" podID="358fb203-2156-4ba6-82be-346b9852c5c8" containerID="991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f" exitCode=0
Jul 16 18:24:18.649136 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649124    2579 generic.go:356] "Generic (PLEG): container finished" podID="358fb203-2156-4ba6-82be-346b9852c5c8" containerID="175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747" exitCode=0
Jul 16 18:24:18.649421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649178    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"
Jul 16 18:24:18.649421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerDied","Data":"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"}
Jul 16 18:24:18.649421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649333    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerDied","Data":"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"}
Jul 16 18:24:18.649421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649361    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b" event={"ID":"358fb203-2156-4ba6-82be-346b9852c5c8","Type":"ContainerDied","Data":"4d22f76524aada8ed702881af8133f3a9bf4e5cf875635ea7507a03aa0ebbbb9"}
Jul 16 18:24:18.649421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.649395    2579 scope.go:117] "RemoveContainer" containerID="991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"
Jul 16 18:24:18.651166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.651134    2579 generic.go:356] "Generic (PLEG): container finished" podID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerID="e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3" exitCode=0
Jul 16 18:24:18.651320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.651178    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"
Jul 16 18:24:18.651320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.651197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" event={"ID":"6fd84871-6341-436c-9fb0-edefbed5f4e6","Type":"ContainerDied","Data":"e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3"}
Jul 16 18:24:18.651320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.651290    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5" event={"ID":"6fd84871-6341-436c-9fb0-edefbed5f4e6","Type":"ContainerDied","Data":"f22d6e38428933de98895e32c7d6835c5371de3d4c296b1c836892f440bc1106"}
Jul 16 18:24:18.660904 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.660878    2579 scope.go:117] "RemoveContainer" containerID="2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"
Jul 16 18:24:18.671631 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.671608    2579 scope.go:117] "RemoveContainer" containerID="175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"
Jul 16 18:24:18.683535 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.683491    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"]
Jul 16 18:24:18.684489 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.684465    2579 scope.go:117] "RemoveContainer" containerID="991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"
Jul 16 18:24:18.685661 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:24:18.685630    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f\": container with ID starting with 991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f not found: ID does not exist" containerID="991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"
Jul 16 18:24:18.685844 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.685672    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"} err="failed to get container status \"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f\": rpc error: code = NotFound desc = could not find container \"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f\": container with ID starting with 991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f not found: ID does not exist"
Jul 16 18:24:18.685844 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.685717    2579 scope.go:117] "RemoveContainer" containerID="2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"
Jul 16 18:24:18.686078 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:24:18.686053    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532\": container with ID starting with 2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532 not found: ID does not exist" containerID="2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"
Jul 16 18:24:18.686159 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.686086    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"} err="failed to get container status \"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532\": rpc error: code = NotFound desc = could not find container \"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532\": container with ID starting with 2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532 not found: ID does not exist"
Jul 16 18:24:18.686159 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.686112    2579 scope.go:117] "RemoveContainer" containerID="175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"
Jul 16 18:24:18.686598 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:24:18.686508    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747\": container with ID starting with 175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747 not found: ID does not exist" containerID="175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"
Jul 16 18:24:18.686598 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.686541    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"} err="failed to get container status \"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747\": rpc error: code = NotFound desc = could not find container \"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747\": container with ID starting with 175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747 not found: ID does not exist"
Jul 16 18:24:18.686598 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.686563    2579 scope.go:117] "RemoveContainer" containerID="991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"
Jul 16 18:24:18.686892 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.686860    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f"} err="failed to get container status \"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f\": rpc error: code = NotFound desc = could not find container \"991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f\": container with ID starting with 991a8fcea0a7d886e9e0e395d3b8ae1fba498499544e493b30175c239ed9a61f not found: ID does not exist"
Jul 16 18:24:18.686969 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.686893    2579 scope.go:117] "RemoveContainer" containerID="2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"
Jul 16 18:24:18.687350 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.687223    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532"} err="failed to get container status \"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532\": rpc error: code = NotFound desc = could not find container \"2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532\": container with ID starting with 2f226ba0917d35c9eaba93438eecee7183f2961b20c52425cb0a0040f2985532 not found: ID does not exist"
Jul 16 18:24:18.687350 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.687249    2579 scope.go:117] "RemoveContainer" containerID="175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"
Jul 16 18:24:18.693397 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.688248    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747"} err="failed to get container status \"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747\": rpc error: code = NotFound desc = could not find container \"175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747\": container with ID starting with 175e070b4ecf4b46bbee0e4d78667f1879be1e9c82ab99766d14103fabdef747 not found: ID does not exist"
Jul 16 18:24:18.693397 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.688305    2579 scope.go:117] "RemoveContainer" containerID="e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3"
Jul 16 18:24:18.693397 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.690588    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-9dcd56f4c-sqbg5"]
Jul 16 18:24:18.707465 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.707400    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"]
Jul 16 18:24:18.717667 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.717624    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-b4685cc89-zfv4b"]
Jul 16 18:24:18.718815 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.718790    2579 scope.go:117] "RemoveContainer" containerID="86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723"
Jul 16 18:24:18.732018 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.731996    2579 scope.go:117] "RemoveContainer" containerID="e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3"
Jul 16 18:24:18.732447 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:24:18.732421    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3\": container with ID starting with e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3 not found: ID does not exist" containerID="e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3"
Jul 16 18:24:18.732551 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.732460    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3"} err="failed to get container status \"e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3\": rpc error: code = NotFound desc = could not find container \"e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3\": container with ID starting with e3791d6579dbbd79b0b1871e0ad4ae2bb0de207e8ec69855260146afea60cff3 not found: ID does not exist"
Jul 16 18:24:18.732551 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.732486    2579 scope.go:117] "RemoveContainer" containerID="86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723"
Jul 16 18:24:18.732880 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:24:18.732805    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723\": container with ID starting with 86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723 not found: ID does not exist" containerID="86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723"
Jul 16 18:24:18.732880 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:18.732845    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723"} err="failed to get container status \"86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723\": rpc error: code = NotFound desc = could not find container \"86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723\": container with ID starting with 86879bf4c2313f421f5ca29ed502baafec5d9603d620880b0a37982b56ebc723 not found: ID does not exist"
Jul 16 18:24:19.103883 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:19.103846    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" path="/var/lib/kubelet/pods/358fb203-2156-4ba6-82be-346b9852c5c8/volumes"
Jul 16 18:24:19.104555 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:19.104533    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" path="/var/lib/kubelet/pods/6fd84871-6341-436c-9fb0-edefbed5f4e6/volumes"
Jul 16 18:24:24.400225 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:24.400177    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:24:25.975182 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975142    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"]
Jul 16 18:24:25.975579 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975542    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="storage-initializer"
Jul 16 18:24:25.975579 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975555    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="storage-initializer"
Jul 16 18:24:25.975579 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975567    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="storage-initializer"
Jul 16 18:24:25.975579 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975573    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="storage-initializer"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975582    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975590    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975624    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975633    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975645    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="llm-d-routing-sidecar"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975651    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="llm-d-routing-sidecar"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975746    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fd84871-6341-436c-9fb0-edefbed5f4e6" containerName="main"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975762    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="main"
Jul 16 18:24:25.975773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.975772    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="358fb203-2156-4ba6-82be-346b9852c5c8" containerName="llm-d-routing-sidecar"
Jul 16 18:24:25.985299 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.978469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:25.985299 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.981589    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 16 18:24:25.994035 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:25.994004    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"]
Jul 16 18:24:26.113638 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v8j8\" (UniqueName: \"kubernetes.io/projected/62bf02e5-9ced-48fe-a41f-820a73d42b94-kube-api-access-6v8j8\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.113848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113697    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.113848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.113848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.113848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.113848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113827    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.114160 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.113930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bf02e5-9ced-48fe-a41f-820a73d42b94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.214620 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214584    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v8j8\" (UniqueName: \"kubernetes.io/projected/62bf02e5-9ced-48fe-a41f-820a73d42b94-kube-api-access-6v8j8\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.214832 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.214832 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214692    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.214832 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214864    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.214919    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bf02e5-9ced-48fe-a41f-820a73d42b94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215190 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.215124    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215292 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.215249    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215386 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.215300    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.215477 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.215450    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.217334 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.217302    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.217549 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.217532    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bf02e5-9ced-48fe-a41f-820a73d42b94-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.224463 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.224442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v8j8\" (UniqueName: \"kubernetes.io/projected/62bf02e5-9ced-48fe-a41f-820a73d42b94-kube-api-access-6v8j8\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.296141 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.296110    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:26.450071 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.450020    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"]
Jul 16 18:24:26.451420 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:24:26.451384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bf02e5_9ced_48fe_a41f_820a73d42b94.slice/crio-4980fe964ec3a240e36b24a8fbc1b28fab1fb1a44965165efbba2cb111234b91 WatchSource:0}: Error finding container 4980fe964ec3a240e36b24a8fbc1b28fab1fb1a44965165efbba2cb111234b91: Status 404 returned error can't find the container with id 4980fe964ec3a240e36b24a8fbc1b28fab1fb1a44965165efbba2cb111234b91
Jul 16 18:24:26.709743 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.709639    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" event={"ID":"62bf02e5-9ced-48fe-a41f-820a73d42b94","Type":"ContainerStarted","Data":"d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f"}
Jul 16 18:24:26.709743 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:26.709701    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" event={"ID":"62bf02e5-9ced-48fe-a41f-820a73d42b94","Type":"ContainerStarted","Data":"4980fe964ec3a240e36b24a8fbc1b28fab1fb1a44965165efbba2cb111234b91"}
Jul 16 18:24:29.730218 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:29.730181    2579 generic.go:356] "Generic (PLEG): container finished" podID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerID="d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f" exitCode=0
Jul 16 18:24:29.730701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:29.730261    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" event={"ID":"62bf02e5-9ced-48fe-a41f-820a73d42b94","Type":"ContainerDied","Data":"d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f"}
Jul 16 18:24:30.739358 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:30.739309    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" event={"ID":"62bf02e5-9ced-48fe-a41f-820a73d42b94","Type":"ContainerStarted","Data":"0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5"}
Jul 16 18:24:30.763992 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:30.763912    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podStartSLOduration=2.645342931 podStartE2EDuration="5.763891317s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.118548386s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:24:30.761487576 +0000 UTC m=+2992.144957545" watchObservedRunningTime="2026-07-16 18:24:30.763891317 +0000 UTC m=+2992.147361253"
Jul 16 18:24:31.296484 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:31.296438    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:34.400560 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:34.400509    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:24:36.297040 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:36.296993    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:24:36.298804 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:36.298760    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:24:44.410783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:44.410745    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:24:44.418698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:44.418671    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:24:46.297289 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:46.297225    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:24:48.444747 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:48.444642    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:24:48.451716 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:48.451686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:24:56.296738 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:24:56.296687    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:06.297246 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:06.297188    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:16.297374 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:16.297325    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:26.297361 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:26.297310    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:36.297488 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:36.297372    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:43.199907 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:43.199858    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"]
Jul 16 18:25:46.297284 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:46.297233    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:56.297204 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:56.297152    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:25:58.237192 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:58.237147    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" containerID="cri-o://df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed" gracePeriod=45
Jul 16 18:25:58.412297 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:58.412237    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 16 18:25:59.207456 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.207426    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:25:59.286921 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.286831    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-dshm\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.286921 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.286892    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tmp-dir\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.286921 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.286918    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-model-cache\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.286955    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-home\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.286985    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksctz\" (UniqueName: \"kubernetes.io/projected/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kube-api-access-ksctz\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287032    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tls-certs\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287092    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kserve-provision-location\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287118    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-cabundle-cert\") pod \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\" (UID: \"e288b40e-5ffa-4fbe-9c1b-e98d835729a0\") "
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287194    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287481    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.287533 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287499    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-home" (OuterVolumeSpecName: "home") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:25:59.287952 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.287558    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:25:59.289463 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.289440    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-dshm" (OuterVolumeSpecName: "dshm") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:25:59.289702 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.289679    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kube-api-access-ksctz" (OuterVolumeSpecName: "kube-api-access-ksctz") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "kube-api-access-ksctz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:25:59.290098 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.290068    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:25:59.304839 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.304810    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:25:59.351987 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.351944    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e288b40e-5ffa-4fbe-9c1b-e98d835729a0" (UID: "e288b40e-5ffa-4fbe-9c1b-e98d835729a0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:25:59.386993 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.386958    2579 generic.go:356] "Generic (PLEG): container finished" podID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerID="df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed" exitCode=0
Jul 16 18:25:59.387180 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387010    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"
Jul 16 18:25:59.387180 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387047    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" event={"ID":"e288b40e-5ffa-4fbe-9c1b-e98d835729a0","Type":"ContainerDied","Data":"df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed"}
Jul 16 18:25:59.387180 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387115    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p" event={"ID":"e288b40e-5ffa-4fbe-9c1b-e98d835729a0","Type":"ContainerDied","Data":"ab7aa86a48c52125833b4e6c66a30eb2e6874fa35d262f8288548e82be872592"}
Jul 16 18:25:59.387180 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387146    2579 scope.go:117] "RemoveContainer" containerID="df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed"
Jul 16 18:25:59.387901 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387877    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.387901 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387902    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.388060 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387917    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksctz\" (UniqueName: \"kubernetes.io/projected/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kube-api-access-ksctz\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.388060 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387933    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.388060 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387948    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.388060 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387959    2579 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-cabundle-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.388060 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.387969    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e288b40e-5ffa-4fbe-9c1b-e98d835729a0-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:25:59.397877 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.397854    2579 scope.go:117] "RemoveContainer" containerID="e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96"
Jul 16 18:25:59.406751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.406734    2579 scope.go:117] "RemoveContainer" containerID="df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed"
Jul 16 18:25:59.407033 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:25:59.407016    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed\": container with ID starting with df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed not found: ID does not exist" containerID="df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed"
Jul 16 18:25:59.407082 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.407044    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed"} err="failed to get container status \"df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed\": rpc error: code = NotFound desc = could not find container \"df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed\": container with ID starting with df6518097244ff14e2eadf7e65ecfac463f4fa7cb75e15beb3a04f3592b539ed not found: ID does not exist"
Jul 16 18:25:59.407082 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.407062    2579 scope.go:117] "RemoveContainer" containerID="e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96"
Jul 16 18:25:59.407372 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:25:59.407260    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96\": container with ID starting with e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96 not found: ID does not exist" containerID="e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96"
Jul 16 18:25:59.407449 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.407380    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96"} err="failed to get container status \"e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96\": rpc error: code = NotFound desc = could not find container \"e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96\": container with ID starting with e0d81388304c40d862af842cd9f94b508f49924872f3a94166f339d73fd07b96 not found: ID does not exist"
Jul 16 18:25:59.418201 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.418167    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"]
Jul 16 18:25:59.429289 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:25:59.429243    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-77c65dc7cf-mfh5p"]
Jul 16 18:26:01.103669 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:01.103632    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" path="/var/lib/kubelet/pods/e288b40e-5ffa-4fbe-9c1b-e98d835729a0/volumes"
Jul 16 18:26:06.297238 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:06.297188    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:26:16.297030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:16.296986    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:26:21.797818 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.797781    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"]
Jul 16 18:26:21.798365 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.798346    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="storage-initializer"
Jul 16 18:26:21.798462 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.798368    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="storage-initializer"
Jul 16 18:26:21.798462 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.798394    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main"
Jul 16 18:26:21.798462 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.798403    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main"
Jul 16 18:26:21.798630 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.798507    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e288b40e-5ffa-4fbe-9c1b-e98d835729a0" containerName="main"
Jul 16 18:26:21.801928 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.801905    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.804903 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.804884    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 16 18:26:21.813987 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.813960    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"]
Jul 16 18:26:21.891416 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-kserve-provision-location\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.891612 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891472    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-tmp-dir\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.891612 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891494    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-model-cache\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.891612 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e822e9f-76b5-4cbd-a743-b858959e3877-tls-certs\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.891612 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891563    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-dshm\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.891821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891633    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-home\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.891821 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.891664    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nq5\" (UniqueName: \"kubernetes.io/projected/5e822e9f-76b5-4cbd-a743-b858959e3877-kube-api-access-c6nq5\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992114 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992066    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-tmp-dir\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992114 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992110    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-model-cache\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992133    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e822e9f-76b5-4cbd-a743-b858959e3877-tls-certs\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-dshm\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-home\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nq5\" (UniqueName: \"kubernetes.io/projected/5e822e9f-76b5-4cbd-a743-b858959e3877-kube-api-access-c6nq5\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992313    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-kserve-provision-location\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992624 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-model-cache\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992673 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-tmp-dir\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992673 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-home\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.992856 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.992836    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-kserve-provision-location\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.994508 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.994483    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-dshm\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:21.994844 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:21.994825    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e822e9f-76b5-4cbd-a743-b858959e3877-tls-certs\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:22.003890 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:22.003864    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nq5\" (UniqueName: \"kubernetes.io/projected/5e822e9f-76b5-4cbd-a743-b858959e3877-kube-api-access-c6nq5\") pod \"tls-verification-test-kserve-57c75fd4f5-wsg7t\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:22.114302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:22.114174    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:22.270710 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:22.270684    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"]
Jul 16 18:26:22.272591 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:26:22.272552    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e822e9f_76b5_4cbd_a743_b858959e3877.slice/crio-fa8bc7f3a926dcf3bf5d7346d95cd98d96c6d5a45cecd475915d16eacbd850c0 WatchSource:0}: Error finding container fa8bc7f3a926dcf3bf5d7346d95cd98d96c6d5a45cecd475915d16eacbd850c0: Status 404 returned error can't find the container with id fa8bc7f3a926dcf3bf5d7346d95cd98d96c6d5a45cecd475915d16eacbd850c0
Jul 16 18:26:22.522592 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:22.522489    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" event={"ID":"5e822e9f-76b5-4cbd-a743-b858959e3877","Type":"ContainerStarted","Data":"efb41e911bcb2bbde9b4255188519847f94f238f1136a437a574f5ca542d02eb"}
Jul 16 18:26:22.522768 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:22.522607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" event={"ID":"5e822e9f-76b5-4cbd-a743-b858959e3877","Type":"ContainerStarted","Data":"fa8bc7f3a926dcf3bf5d7346d95cd98d96c6d5a45cecd475915d16eacbd850c0"}
Jul 16 18:26:25.547544 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:25.547513    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerID="efb41e911bcb2bbde9b4255188519847f94f238f1136a437a574f5ca542d02eb" exitCode=0
Jul 16 18:26:25.547938 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:25.547585    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" event={"ID":"5e822e9f-76b5-4cbd-a743-b858959e3877","Type":"ContainerDied","Data":"efb41e911bcb2bbde9b4255188519847f94f238f1136a437a574f5ca542d02eb"}
Jul 16 18:26:26.297491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:26.297416    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:26:26.557155 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:26.557054    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" event={"ID":"5e822e9f-76b5-4cbd-a743-b858959e3877","Type":"ContainerStarted","Data":"3b71f40872f35707e1df5517cec7f9382b971b36c31be0f6f5915de7d0a36495"}
Jul 16 18:26:26.587476 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:26.587406    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podStartSLOduration=2.568125475 podStartE2EDuration="5.587381056s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.019255581s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:26:26.582456685 +0000 UTC m=+3107.965926621" watchObservedRunningTime="2026-07-16 18:26:26.587381056 +0000 UTC m=+3107.970850990"
Jul 16 18:26:27.114491 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:27.114456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:32.114769 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:32.114729    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:26:32.116749 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:32.116717    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:26:36.297530 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:36.297485    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:26:42.115134 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:42.115067    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:26:46.307727 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:46.307690    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:26:46.316087 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:46.316055    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:26:50.799278 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:50.799223    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"]
Jul 16 18:26:52.115431 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:26:52.115378    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:02.115166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:02.115068    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:05.822937 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:05.822866    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" containerID="cri-o://0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5" gracePeriod=45
Jul 16 18:27:06.308464 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.308423    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/health\": dial tcp 10.133.0.73:8000: connect: connection refused"
Jul 16 18:27:06.573711 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.573684    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:27:06.639463 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v8j8\" (UniqueName: \"kubernetes.io/projected/62bf02e5-9ced-48fe-a41f-820a73d42b94-kube-api-access-6v8j8\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.639638 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639492    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-dshm\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.639638 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639539    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-model-cache\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.639638 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639557    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-kserve-provision-location\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.639638 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bf02e5-9ced-48fe-a41f-820a73d42b94-tls-certs\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.639868 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639643    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-tmp-dir\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.639868 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.639804    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-model-cache" (OuterVolumeSpecName: "model-cache") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:27:06.640249 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.640185    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-home\") pod \"62bf02e5-9ced-48fe-a41f-820a73d42b94\" (UID: \"62bf02e5-9ced-48fe-a41f-820a73d42b94\") "
Jul 16 18:27:06.640469 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.640441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-home" (OuterVolumeSpecName: "home") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:27:06.640631 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.640562    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.640631 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.640633    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.641970 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.641930    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-dshm" (OuterVolumeSpecName: "dshm") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:27:06.642184 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.642151    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf02e5-9ced-48fe-a41f-820a73d42b94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:27:06.642184 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.642170    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bf02e5-9ced-48fe-a41f-820a73d42b94-kube-api-access-6v8j8" (OuterVolumeSpecName: "kube-api-access-6v8j8") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "kube-api-access-6v8j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:27:06.653764 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.653733    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:27:06.706006 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.705947    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62bf02e5-9ced-48fe-a41f-820a73d42b94" (UID: "62bf02e5-9ced-48fe-a41f-820a73d42b94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:27:06.741212 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.741180    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62bf02e5-9ced-48fe-a41f-820a73d42b94-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.741212 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.741210    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.741212 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.741220    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v8j8\" (UniqueName: \"kubernetes.io/projected/62bf02e5-9ced-48fe-a41f-820a73d42b94-kube-api-access-6v8j8\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.741469 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.741229    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.741469 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.741238    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62bf02e5-9ced-48fe-a41f-820a73d42b94-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:27:06.795375 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.795329    2579 generic.go:356] "Generic (PLEG): container finished" podID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerID="0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5" exitCode=0
Jul 16 18:27:06.795375 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.795381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"
Jul 16 18:27:06.795610 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.795412    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" event={"ID":"62bf02e5-9ced-48fe-a41f-820a73d42b94","Type":"ContainerDied","Data":"0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5"}
Jul 16 18:27:06.795610 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.795468    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn" event={"ID":"62bf02e5-9ced-48fe-a41f-820a73d42b94","Type":"ContainerDied","Data":"4980fe964ec3a240e36b24a8fbc1b28fab1fb1a44965165efbba2cb111234b91"}
Jul 16 18:27:06.795610 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.795497    2579 scope.go:117] "RemoveContainer" containerID="0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5"
Jul 16 18:27:06.805353 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.805334    2579 scope.go:117] "RemoveContainer" containerID="d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f"
Jul 16 18:27:06.814610 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.814586    2579 scope.go:117] "RemoveContainer" containerID="0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5"
Jul 16 18:27:06.814883 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:27:06.814856    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5\": container with ID starting with 0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5 not found: ID does not exist" containerID="0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5"
Jul 16 18:27:06.814949 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.814894    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5"} err="failed to get container status \"0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5\": rpc error: code = NotFound desc = could not find container \"0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5\": container with ID starting with 0b770930a6cdd3061bd148823d9b06ecfcd18fda362bc251d7de5537a663bcc5 not found: ID does not exist"
Jul 16 18:27:06.814949 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.814917    2579 scope.go:117] "RemoveContainer" containerID="d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f"
Jul 16 18:27:06.815214 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:27:06.815192    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f\": container with ID starting with d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f not found: ID does not exist" containerID="d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f"
Jul 16 18:27:06.815316 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.815220    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f"} err="failed to get container status \"d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f\": rpc error: code = NotFound desc = could not find container \"d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f\": container with ID starting with d5f37d698acd33d48c11e56b05a926e98e05697490438c0d7fe5868de06ccf9f not found: ID does not exist"
Jul 16 18:27:06.823860 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.823837    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"]
Jul 16 18:27:06.828406 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:06.828383    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-7c7fd556fb9fphn"]
Jul 16 18:27:07.102923 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:07.102837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" path="/var/lib/kubelet/pods/62bf02e5-9ced-48fe-a41f-820a73d42b94/volumes"
Jul 16 18:27:12.115155 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:12.115113    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:22.115581 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:22.115533    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:28.294275 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.294225    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 18:27:28.294718 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.294692    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="storage-initializer"
Jul 16 18:27:28.294718 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.294718    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="storage-initializer"
Jul 16 18:27:28.294931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.294745    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main"
Jul 16 18:27:28.294931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.294754    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main"
Jul 16 18:27:28.294931 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.294855    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62bf02e5-9ced-48fe-a41f-820a73d42b94" containerName="main"
Jul 16 18:27:28.298176 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.298157    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.301635 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.301600    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-2tczx\""
Jul 16 18:27:28.301804 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.301783    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 16 18:27:28.311182 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.311155    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 18:27:28.345299 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.345244    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 18:27:28.349786 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.349761    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.375365 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.375322    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 18:27:28.446791 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446740    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe7dca-0611-4c4a-993a-920b12fff02c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.446791 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446865    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446958    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.446988    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3295fe8-166b-4de6-8ce5-28c68eacf297-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.447004    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.447043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.447023    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj9kx\" (UniqueName: \"kubernetes.io/projected/e3295fe8-166b-4de6-8ce5-28c68eacf297-kube-api-access-sj9kx\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.447430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.447120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb8ct\" (UniqueName: \"kubernetes.io/projected/09fe7dca-0611-4c4a-993a-920b12fff02c-kube-api-access-mb8ct\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.447430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.447164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.447430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.447198    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.447430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.447227    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.552101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552013    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.552373 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.552512 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.552637 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552623    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe7dca-0611-4c4a-993a-920b12fff02c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.552752 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.552869 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552843    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.552953 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.553048 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552432    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.553112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.552681    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.553112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.553068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.553300 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.553257    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.553912 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.553893    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.554063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554048    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.554186 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554173    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.554360 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3295fe8-166b-4de6-8ce5-28c68eacf297-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.554495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.554495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj9kx\" (UniqueName: \"kubernetes.io/projected/e3295fe8-166b-4de6-8ce5-28c68eacf297-kube-api-access-sj9kx\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.554495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.554495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554454    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb8ct\" (UniqueName: \"kubernetes.io/projected/09fe7dca-0611-4c4a-993a-920b12fff02c-kube-api-access-mb8ct\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.554728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.553625    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.554928 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554903    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.555005 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.554981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.556790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.556742    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe7dca-0611-4c4a-993a-920b12fff02c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.556916 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.556841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.557070 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.557048    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.557648 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.557617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3295fe8-166b-4de6-8ce5-28c68eacf297-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.564411 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.564360    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb8ct\" (UniqueName: \"kubernetes.io/projected/09fe7dca-0611-4c4a-993a-920b12fff02c-kube-api-access-mb8ct\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.564831 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.564812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj9kx\" (UniqueName: \"kubernetes.io/projected/e3295fe8-166b-4de6-8ce5-28c68eacf297-kube-api-access-sj9kx\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.609469 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.609430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:28.660295 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.660191    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:27:28.768204 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.768158    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 18:27:28.773502 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:27:28.773465    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09fe7dca_0611_4c4a_993a_920b12fff02c.slice/crio-c33f5a15176f432943aaf574aa08e997719e3f6cc80e41a097cdba85d219b125 WatchSource:0}: Error finding container c33f5a15176f432943aaf574aa08e997719e3f6cc80e41a097cdba85d219b125: Status 404 returned error can't find the container with id c33f5a15176f432943aaf574aa08e997719e3f6cc80e41a097cdba85d219b125
Jul 16 18:27:28.827593 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.825675    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 18:27:28.829407 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:27:28.829375    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3295fe8_166b_4de6_8ce5_28c68eacf297.slice/crio-dc36ad71878148cd3d02dcdfc18da7c7135c4f18f3fb5c28c9223128de57a13e WatchSource:0}: Error finding container dc36ad71878148cd3d02dcdfc18da7c7135c4f18f3fb5c28c9223128de57a13e: Status 404 returned error can't find the container with id dc36ad71878148cd3d02dcdfc18da7c7135c4f18f3fb5c28c9223128de57a13e
Jul 16 18:27:28.929193 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.929116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e3295fe8-166b-4de6-8ce5-28c68eacf297","Type":"ContainerStarted","Data":"61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131"}
Jul 16 18:27:28.929700 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.929673    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e3295fe8-166b-4de6-8ce5-28c68eacf297","Type":"ContainerStarted","Data":"dc36ad71878148cd3d02dcdfc18da7c7135c4f18f3fb5c28c9223128de57a13e"}
Jul 16 18:27:28.930985 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.930957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09fe7dca-0611-4c4a-993a-920b12fff02c","Type":"ContainerStarted","Data":"8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef"}
Jul 16 18:27:28.931119 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:28.931008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09fe7dca-0611-4c4a-993a-920b12fff02c","Type":"ContainerStarted","Data":"c33f5a15176f432943aaf574aa08e997719e3f6cc80e41a097cdba85d219b125"}
Jul 16 18:27:32.115524 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:32.115480    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:32.969107 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:32.969042    2579 generic.go:356] "Generic (PLEG): container finished" podID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerID="61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131" exitCode=0
Jul 16 18:27:32.969406 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:32.969125    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e3295fe8-166b-4de6-8ce5-28c68eacf297","Type":"ContainerDied","Data":"61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131"}
Jul 16 18:27:32.971687 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:32.971654    2579 generic.go:356] "Generic (PLEG): container finished" podID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerID="8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef" exitCode=0
Jul 16 18:27:32.971858 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:32.971710    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09fe7dca-0611-4c4a-993a-920b12fff02c","Type":"ContainerDied","Data":"8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef"}
Jul 16 18:27:33.981034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:33.980985    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e3295fe8-166b-4de6-8ce5-28c68eacf297","Type":"ContainerStarted","Data":"17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f"}
Jul 16 18:27:33.983649 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:33.983619    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09fe7dca-0611-4c4a-993a-920b12fff02c","Type":"ContainerStarted","Data":"2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635"}
Jul 16 18:27:34.026895 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:34.026826    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.7074780389999997 podStartE2EDuration="6.026804186s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.319326147s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:27:34.004643707 +0000 UTC m=+3175.388113732" watchObservedRunningTime="2026-07-16 18:27:34.026804186 +0000 UTC m=+3175.410274124"
Jul 16 18:27:34.028384 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:34.028335    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.838780431 podStartE2EDuration="6.028317289s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.189536858s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:27:34.024453462 +0000 UTC m=+3175.407923408" watchObservedRunningTime="2026-07-16 18:27:34.028317289 +0000 UTC m=+3175.411787224"
Jul 16 18:27:34.609935 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:34.609896    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:38.610426 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:38.610386    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:27:38.612566 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:38.612530    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:27:42.115353 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:42.115296    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:48.610206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:48.610145    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:27:52.114911 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:52.114872    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:27:58.610043 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:27:58.609995    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:28:02.114672 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:02.114620    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:28:08.610246 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:08.610192    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:28:12.114772 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:12.114707    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:28:18.610509 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:18.610464    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:28:22.130627 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:22.130592    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:28:22.138606 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:22.138573    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:28:27.197205 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:27.197147    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"]
Jul 16 18:28:28.610298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:28.610238    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:28:38.610176 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:38.610072    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:28:42.233396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:42.233333    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" containerID="cri-o://3b71f40872f35707e1df5517cec7f9382b971b36c31be0f6f5915de7d0a36495" gracePeriod=45
Jul 16 18:28:43.132077 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.132035    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 16 18:28:43.461966 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.461928    2579 generic.go:356] "Generic (PLEG): container finished" podID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerID="3b71f40872f35707e1df5517cec7f9382b971b36c31be0f6f5915de7d0a36495" exitCode=0
Jul 16 18:28:43.462477 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.462016    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" event={"ID":"5e822e9f-76b5-4cbd-a743-b858959e3877","Type":"ContainerDied","Data":"3b71f40872f35707e1df5517cec7f9382b971b36c31be0f6f5915de7d0a36495"}
Jul 16 18:28:43.493179 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.493151    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:28:43.529016 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.528982    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e822e9f-76b5-4cbd-a743-b858959e3877-tls-certs\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529016 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529017    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6nq5\" (UniqueName: \"kubernetes.io/projected/5e822e9f-76b5-4cbd-a743-b858959e3877-kube-api-access-c6nq5\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529046    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-kserve-provision-location\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529073    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-dshm\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529098    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-model-cache\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529135    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-tmp-dir\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529535 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529371    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-home\") pod \"5e822e9f-76b5-4cbd-a743-b858959e3877\" (UID: \"5e822e9f-76b5-4cbd-a743-b858959e3877\") "
Jul 16 18:28:43.529535 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:43.529709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.529687    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.530064 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.530035    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-home" (OuterVolumeSpecName: "home") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:43.531648 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.531625    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-dshm" (OuterVolumeSpecName: "dshm") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:43.532080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.532054    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e822e9f-76b5-4cbd-a743-b858959e3877-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:28:43.532304 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.532252    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e822e9f-76b5-4cbd-a743-b858959e3877-kube-api-access-c6nq5" (OuterVolumeSpecName: "kube-api-access-c6nq5") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "kube-api-access-c6nq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:28:43.542141 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.542106    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:43.631326 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.631287    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.631326 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.631321    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e822e9f-76b5-4cbd-a743-b858959e3877-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.631326 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.631332    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nq5\" (UniqueName: \"kubernetes.io/projected/5e822e9f-76b5-4cbd-a743-b858959e3877-kube-api-access-c6nq5\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.631581 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.631347    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.631581 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.631356    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:43.635694 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.635659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5e822e9f-76b5-4cbd-a743-b858959e3877" (UID: "5e822e9f-76b5-4cbd-a743-b858959e3877"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:28:43.732036 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:43.731941    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5e822e9f-76b5-4cbd-a743-b858959e3877-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:28:44.469535 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:44.469505    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"
Jul 16 18:28:44.470162 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:44.469532    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t" event={"ID":"5e822e9f-76b5-4cbd-a743-b858959e3877","Type":"ContainerDied","Data":"fa8bc7f3a926dcf3bf5d7346d95cd98d96c6d5a45cecd475915d16eacbd850c0"}
Jul 16 18:28:44.470162 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:44.469611    2579 scope.go:117] "RemoveContainer" containerID="3b71f40872f35707e1df5517cec7f9382b971b36c31be0f6f5915de7d0a36495"
Jul 16 18:28:44.479497 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:44.479477    2579 scope.go:117] "RemoveContainer" containerID="efb41e911bcb2bbde9b4255188519847f94f238f1136a437a574f5ca542d02eb"
Jul 16 18:28:44.497171 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:44.497143    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"]
Jul 16 18:28:44.500588 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:44.500560    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-57c75fd4f5-wsg7t"]
Jul 16 18:28:45.104892 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:45.104850    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" path="/var/lib/kubelet/pods/5e822e9f-76b5-4cbd-a743-b858959e3877/volumes"
Jul 16 18:28:48.610461 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:48.610422    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:28:50.393784 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.393750    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"]
Jul 16 18:28:50.394174 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.394149    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main"
Jul 16 18:28:50.394174 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.394161    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main"
Jul 16 18:28:50.394261 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.394179    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="storage-initializer"
Jul 16 18:28:50.394261 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.394187    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="storage-initializer"
Jul 16 18:28:50.394261 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.394252    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e822e9f-76b5-4cbd-a743-b858959e3877" containerName="main"
Jul 16 18:28:50.399279 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.399252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.403647 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.403625    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 16 18:28:50.409866 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.409831    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"]
Jul 16 18:28:50.493015 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.492979    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-dshm\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.493209 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.493110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-home\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.493209 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.493164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-tmp-dir\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.493209 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.493201    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4987bc4b-6330-4753-a140-259cb3e2af12-tls-certs\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.493398 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.493356    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-model-cache\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.493398 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.493381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.493398 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.493397    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kft\" (UniqueName: \"kubernetes.io/projected/4987bc4b-6330-4753-a140-259cb3e2af12-kube-api-access-k7kft\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594330 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-home\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-tmp-dir\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4987bc4b-6330-4753-a140-259cb3e2af12-tls-certs\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-model-cache\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594473    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594522 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594495    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kft\" (UniqueName: \"kubernetes.io/projected/4987bc4b-6330-4753-a140-259cb3e2af12-kube-api-access-k7kft\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594873 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-dshm\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594873 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-tmp-dir\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594867    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.594986 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.594878    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-model-cache\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.595067 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.595047    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-home\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.596749 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.596727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-dshm\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.597063 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.597044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4987bc4b-6330-4753-a140-259cb3e2af12-tls-certs\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.603933 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.603914    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kft\" (UniqueName: \"kubernetes.io/projected/4987bc4b-6330-4753-a140-259cb3e2af12-kube-api-access-k7kft\") pod \"prestop-hook-test-kserve-5c7d67c6bc-nm7qd\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.711686 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.711650    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:50.874982 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:50.874953    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"]
Jul 16 18:28:50.878068 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:28:50.878041    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4987bc4b_6330_4753_a140_259cb3e2af12.slice/crio-457b027efef87eb81b8d3f1fed4ed07df7bbbaced14993510a17d2085af62c82 WatchSource:0}: Error finding container 457b027efef87eb81b8d3f1fed4ed07df7bbbaced14993510a17d2085af62c82: Status 404 returned error can't find the container with id 457b027efef87eb81b8d3f1fed4ed07df7bbbaced14993510a17d2085af62c82
Jul 16 18:28:51.514308 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:51.514242    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" event={"ID":"4987bc4b-6330-4753-a140-259cb3e2af12","Type":"ContainerStarted","Data":"c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c"}
Jul 16 18:28:51.514803 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:51.514339    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" event={"ID":"4987bc4b-6330-4753-a140-259cb3e2af12","Type":"ContainerStarted","Data":"457b027efef87eb81b8d3f1fed4ed07df7bbbaced14993510a17d2085af62c82"}
Jul 16 18:28:54.534564 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:54.534461    2579 generic.go:356] "Generic (PLEG): container finished" podID="4987bc4b-6330-4753-a140-259cb3e2af12" containerID="c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c" exitCode=0
Jul 16 18:28:54.534564 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:54.534538    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" event={"ID":"4987bc4b-6330-4753-a140-259cb3e2af12","Type":"ContainerDied","Data":"c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c"}
Jul 16 18:28:55.543713 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:55.543671    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" event={"ID":"4987bc4b-6330-4753-a140-259cb3e2af12","Type":"ContainerStarted","Data":"2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0"}
Jul 16 18:28:55.573080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:55.573010    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podStartSLOduration=2.347853526 podStartE2EDuration="5.572984234s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.225130708s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:28:55.568842866 +0000 UTC m=+3256.952312797" watchObservedRunningTime="2026-07-16 18:28:55.572984234 +0000 UTC m=+3256.956454169"
Jul 16 18:28:55.711976 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:55.711936    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:28:58.610797 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:28:58.610745    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:29:00.711883 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:00.711842    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:29:00.713823 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:00.713782    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:29:08.610941 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:08.610874    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:29:10.712630 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:10.712590    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:29:18.610388 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:18.610332    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:29:20.712885 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:20.712841    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:29:28.610387 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:28.610340    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:29:30.712396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:30.712338    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:29:38.610740 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:38.610688    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:29:40.712807 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:40.712758    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:29:48.504358 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:48.504319    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:29:48.510793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:48.510771    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:29:48.610686 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:48.610640    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:29:50.712359 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:50.712311    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:29:58.620200 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:58.620160    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:29:58.627677 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:29:58.627650    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:30:00.712313 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:00.712252    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:30:10.712550 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:10.712501    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:30:20.713051 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:20.713004    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:30:30.712488 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:30.712442    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:30:40.712370 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:40.712321    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:30:50.721855 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:50.721820    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:30:50.730834 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:50.730805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:30:53.377670 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.377632    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"]
Jul 16 18:30:53.407331 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.407231    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"]
Jul 16 18:30:53.410694 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.410678    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.425605 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.425579    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"]
Jul 16 18:30:53.519530 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.519485    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-model-cache\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.519837 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.519818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tmp-dir\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.519969 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.519952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.520144 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.520127    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-home\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.520410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.520378    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tls-certs\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.520493 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.520450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvbcz\" (UniqueName: \"kubernetes.io/projected/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kube-api-access-qvbcz\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.520551 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.520522    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-dshm\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621173 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621137    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tmp-dir\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621173 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-home\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621432    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tls-certs\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621575 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621468    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvbcz\" (UniqueName: \"kubernetes.io/projected/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kube-api-access-qvbcz\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621575 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621547    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-dshm\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621638    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kserve-provision-location\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tmp-dir\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621719    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-home\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-model-cache\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.621965 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.621945    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-model-cache\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.623752 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.623727    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-dshm\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.624140 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.624116    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tls-certs\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.631365 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.631344    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvbcz\" (UniqueName: \"kubernetes.io/projected/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kube-api-access-qvbcz\") pod \"prestop-hook-test-kserve-5c7d67c6bc-dzgk6\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.720790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.720690    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:30:53.865293 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:53.865246    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"]
Jul 16 18:30:53.866486 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:30:53.866447    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c161feb_c0c5_4ff4_89cc_0e2eb9a4935c.slice/crio-683558d9ada5ed1d86918a738d41996162dd683b05caeba9be587dabdd62115d WatchSource:0}: Error finding container 683558d9ada5ed1d86918a738d41996162dd683b05caeba9be587dabdd62115d: Status 404 returned error can't find the container with id 683558d9ada5ed1d86918a738d41996162dd683b05caeba9be587dabdd62115d
Jul 16 18:30:54.267207 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:54.267166    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" event={"ID":"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c","Type":"ContainerStarted","Data":"27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09"}
Jul 16 18:30:54.267420 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:54.267252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" event={"ID":"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c","Type":"ContainerStarted","Data":"683558d9ada5ed1d86918a738d41996162dd683b05caeba9be587dabdd62115d"}
Jul 16 18:30:57.285018 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:57.284987    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerID="27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09" exitCode=0
Jul 16 18:30:57.285398 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:57.285039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" event={"ID":"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c","Type":"ContainerDied","Data":"27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09"}
Jul 16 18:30:58.296069 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:58.296033    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" event={"ID":"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c","Type":"ContainerStarted","Data":"e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861"}
Jul 16 18:30:58.325161 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:58.325105    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" podStartSLOduration=2.11357266 podStartE2EDuration="5.325087026s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.211514366s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:30:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:30:58.321300908 +0000 UTC m=+3379.704770844" watchObservedRunningTime="2026-07-16 18:30:58.325087026 +0000 UTC m=+3379.708556959"
Jul 16 18:30:58.721778 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:30:58.721691    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:31:03.721817 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:03.721771    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:31:03.723719 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:03.723682    2579 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 16 18:31:08.401101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:08.401057    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" containerID="cri-o://2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0" gracePeriod=45
Jul 16 18:31:08.722777 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:08.722674    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 16 18:31:09.292858 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.292830    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:31:09.362776 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.362740    2579 generic.go:356] "Generic (PLEG): container finished" podID="4987bc4b-6330-4753-a140-259cb3e2af12" containerID="2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0" exitCode=0
Jul 16 18:31:09.362975 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.362802    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"
Jul 16 18:31:09.362975 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.362819    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" event={"ID":"4987bc4b-6330-4753-a140-259cb3e2af12","Type":"ContainerDied","Data":"2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0"}
Jul 16 18:31:09.362975 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.362890    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd" event={"ID":"4987bc4b-6330-4753-a140-259cb3e2af12","Type":"ContainerDied","Data":"457b027efef87eb81b8d3f1fed4ed07df7bbbaced14993510a17d2085af62c82"}
Jul 16 18:31:09.362975 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.362924    2579 scope.go:117] "RemoveContainer" containerID="2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0"
Jul 16 18:31:09.373175 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.373150    2579 scope.go:117] "RemoveContainer" containerID="c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c"
Jul 16 18:31:09.383486 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.383460    2579 scope.go:117] "RemoveContainer" containerID="2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0"
Jul 16 18:31:09.383801 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:31:09.383782    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0\": container with ID starting with 2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0 not found: ID does not exist" containerID="2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0"
Jul 16 18:31:09.383898 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.383814    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0"} err="failed to get container status \"2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0\": rpc error: code = NotFound desc = could not find container \"2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0\": container with ID starting with 2a5c51103bd4c0b346806a3ca75cb1a22b0f4001bcedc9797d75a6c161f908d0 not found: ID does not exist"
Jul 16 18:31:09.383898 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.383842    2579 scope.go:117] "RemoveContainer" containerID="c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c"
Jul 16 18:31:09.384180 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:31:09.384159    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c\": container with ID starting with c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c not found: ID does not exist" containerID="c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c"
Jul 16 18:31:09.384222 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.384187    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c"} err="failed to get container status \"c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c\": rpc error: code = NotFound desc = could not find container \"c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c\": container with ID starting with c7a6acf62a6a84d35ffae2074a4a413126e96eb65c76e201dd60f2d5f989a66c not found: ID does not exist"
Jul 16 18:31:09.386488 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4987bc4b-6330-4753-a140-259cb3e2af12-tls-certs\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386567 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386508    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-dshm\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386567 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386553    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-model-cache\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-kserve-provision-location\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386603    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-home\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-tmp-dir\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386835 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386738    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7kft\" (UniqueName: \"kubernetes.io/projected/4987bc4b-6330-4753-a140-259cb3e2af12-kube-api-access-k7kft\") pod \"4987bc4b-6330-4753-a140-259cb3e2af12\" (UID: \"4987bc4b-6330-4753-a140-259cb3e2af12\") "
Jul 16 18:31:09.386974 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.386932    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-model-cache" (OuterVolumeSpecName: "model-cache") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:09.387101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.387077    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.387377 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.387350    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-home" (OuterVolumeSpecName: "home") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:09.388977 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.388905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-dshm" (OuterVolumeSpecName: "dshm") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:09.389254 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.389236    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4987bc4b-6330-4753-a140-259cb3e2af12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:31:09.389524 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.389501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4987bc4b-6330-4753-a140-259cb3e2af12-kube-api-access-k7kft" (OuterVolumeSpecName: "kube-api-access-k7kft") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "kube-api-access-k7kft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:31:09.404992 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.404921    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:09.440256 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.440216    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4987bc4b-6330-4753-a140-259cb3e2af12" (UID: "4987bc4b-6330-4753-a140-259cb3e2af12"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:09.487970 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.487934    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.487970 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.487964    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.488175 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.487995    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.488175 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.488003    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kft\" (UniqueName: \"kubernetes.io/projected/4987bc4b-6330-4753-a140-259cb3e2af12-kube-api-access-k7kft\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.488175 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.488013    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4987bc4b-6330-4753-a140-259cb3e2af12-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.488175 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.488022    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4987bc4b-6330-4753-a140-259cb3e2af12-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:09.693470 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.693437    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"]
Jul 16 18:31:09.701882 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:09.701848    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-nm7qd"]
Jul 16 18:31:11.104025 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:11.103978    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" path="/var/lib/kubelet/pods/4987bc4b-6330-4753-a140-259cb3e2af12/volumes"
Jul 16 18:31:11.798759 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:11.798719    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"]
Jul 16 18:31:18.097521 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:18.097486    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 18:31:26.823811 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:26.823760    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="main" containerID="cri-o://e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861" gracePeriod=45
Jul 16 18:31:33.121631 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:33.121506    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" containerID="cri-o://2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635" gracePeriod=45
Jul 16 18:31:33.621670 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:33.621621    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 16 18:31:33.891797 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:33.891771    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:31:34.023401 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023360    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb8ct\" (UniqueName: \"kubernetes.io/projected/09fe7dca-0611-4c4a-993a-920b12fff02c-kube-api-access-mb8ct\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.023591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023430    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe7dca-0611-4c4a-993a-920b12fff02c-tls-certs\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.023591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023483    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-tmp-dir\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.023591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-dshm\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.023591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023545    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-home\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.023591 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023570    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-kserve-provision-location\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.023934 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.023631    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-model-cache\") pod \"09fe7dca-0611-4c4a-993a-920b12fff02c\" (UID: \"09fe7dca-0611-4c4a-993a-920b12fff02c\") "
Jul 16 18:31:34.024143 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.024093    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-model-cache" (OuterVolumeSpecName: "model-cache") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:34.024332 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.024235    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-home" (OuterVolumeSpecName: "home") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:34.026067 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.026031    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09fe7dca-0611-4c4a-993a-920b12fff02c-kube-api-access-mb8ct" (OuterVolumeSpecName: "kube-api-access-mb8ct") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "kube-api-access-mb8ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:31:34.026199 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.026135    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-dshm" (OuterVolumeSpecName: "dshm") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:34.027505 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.027476    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09fe7dca-0611-4c4a-993a-920b12fff02c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:31:34.036689 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.036659    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:34.086801 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.086729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09fe7dca-0611-4c4a-993a-920b12fff02c" (UID: "09fe7dca-0611-4c4a-993a-920b12fff02c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:34.125239 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125196    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.125239 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125240    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.125701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125255    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.125701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125294    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb8ct\" (UniqueName: \"kubernetes.io/projected/09fe7dca-0611-4c4a-993a-920b12fff02c-kube-api-access-mb8ct\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.125701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125308    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09fe7dca-0611-4c4a-993a-920b12fff02c-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.125701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125322    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.125701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.125333    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09fe7dca-0611-4c4a-993a-920b12fff02c-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:34.522468 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.522368    2579 generic.go:356] "Generic (PLEG): container finished" podID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerID="2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635" exitCode=0
Jul 16 18:31:34.522468 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.522440    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 16 18:31:34.522680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.522443    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09fe7dca-0611-4c4a-993a-920b12fff02c","Type":"ContainerDied","Data":"2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635"}
Jul 16 18:31:34.522680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.522577    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"09fe7dca-0611-4c4a-993a-920b12fff02c","Type":"ContainerDied","Data":"c33f5a15176f432943aaf574aa08e997719e3f6cc80e41a097cdba85d219b125"}
Jul 16 18:31:34.522680 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.522607    2579 scope.go:117] "RemoveContainer" containerID="2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635"
Jul 16 18:31:34.534546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.534530    2579 scope.go:117] "RemoveContainer" containerID="8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef"
Jul 16 18:31:34.543147 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.543127    2579 scope.go:117] "RemoveContainer" containerID="2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635"
Jul 16 18:31:34.543437 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:31:34.543418    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635\": container with ID starting with 2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635 not found: ID does not exist" containerID="2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635"
Jul 16 18:31:34.543504 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.543447    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635"} err="failed to get container status \"2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635\": rpc error: code = NotFound desc = could not find container \"2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635\": container with ID starting with 2d4600bed51cec7ee5d0297fc70f1c3715a7a811f3f04154bc8650ffc1d2b635 not found: ID does not exist"
Jul 16 18:31:34.543504 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.543466    2579 scope.go:117] "RemoveContainer" containerID="8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef"
Jul 16 18:31:34.543728 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:31:34.543711    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef\": container with ID starting with 8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef not found: ID does not exist" containerID="8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef"
Jul 16 18:31:34.543783 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.543735    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef"} err="failed to get container status \"8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef\": rpc error: code = NotFound desc = could not find container \"8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef\": container with ID starting with 8549eca607ad83ec92409c17370717575fcd2e493ade44d6265885737339d0ef not found: ID does not exist"
Jul 16 18:31:34.549118 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.549095    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 18:31:34.554346 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.554326    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 16 18:31:34.584213 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:34.584180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 18:31:35.103225 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:35.103188    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" path="/var/lib/kubelet/pods/09fe7dca-0611-4c4a-993a-920b12fff02c/volumes"
Jul 16 18:31:49.607140 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:49.607065    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerName="main" containerID="cri-o://17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f" gracePeriod=45
Jul 16 18:31:50.273065 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.273037    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:31:50.380241 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380143    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3295fe8-166b-4de6-8ce5-28c68eacf297-tls-certs\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380241 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-model-cache\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380299    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-kserve-provision-location\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380339    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj9kx\" (UniqueName: \"kubernetes.io/projected/e3295fe8-166b-4de6-8ce5-28c68eacf297-kube-api-access-sj9kx\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-dshm\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380520 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380389    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-home\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380744 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380420    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-tmp-dir\") pod \"e3295fe8-166b-4de6-8ce5-28c68eacf297\" (UID: \"e3295fe8-166b-4de6-8ce5-28c68eacf297\") "
Jul 16 18:31:50.380744 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.380578    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-model-cache" (OuterVolumeSpecName: "model-cache") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:50.381145 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.381122    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.381380 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.381353    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-home" (OuterVolumeSpecName: "home") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:50.382784 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.382753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3295fe8-166b-4de6-8ce5-28c68eacf297-kube-api-access-sj9kx" (OuterVolumeSpecName: "kube-api-access-sj9kx") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "kube-api-access-sj9kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:31:50.383432 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.383395    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3295fe8-166b-4de6-8ce5-28c68eacf297-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:31:50.383588 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.383566    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-dshm" (OuterVolumeSpecName: "dshm") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:50.393119 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.393089    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:50.439793 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.439729    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e3295fe8-166b-4de6-8ce5-28c68eacf297" (UID: "e3295fe8-166b-4de6-8ce5-28c68eacf297"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:31:50.482196 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.482159    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.482196 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.482191    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj9kx\" (UniqueName: \"kubernetes.io/projected/e3295fe8-166b-4de6-8ce5-28c68eacf297-kube-api-access-sj9kx\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.482196 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.482201    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.482430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.482209    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.482430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.482218    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3295fe8-166b-4de6-8ce5-28c68eacf297-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.482430 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.482225    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3295fe8-166b-4de6-8ce5-28c68eacf297-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:31:50.622453 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.622413    2579 generic.go:356] "Generic (PLEG): container finished" podID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerID="17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f" exitCode=0
Jul 16 18:31:50.622881 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.622466    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 16 18:31:50.622881 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.622481    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e3295fe8-166b-4de6-8ce5-28c68eacf297","Type":"ContainerDied","Data":"17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f"}
Jul 16 18:31:50.622881 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.622539    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"e3295fe8-166b-4de6-8ce5-28c68eacf297","Type":"ContainerDied","Data":"dc36ad71878148cd3d02dcdfc18da7c7135c4f18f3fb5c28c9223128de57a13e"}
Jul 16 18:31:50.622881 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.622570    2579 scope.go:117] "RemoveContainer" containerID="17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f"
Jul 16 18:31:50.632292 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.632254    2579 scope.go:117] "RemoveContainer" containerID="61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131"
Jul 16 18:31:50.642348 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.642323    2579 scope.go:117] "RemoveContainer" containerID="17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f"
Jul 16 18:31:50.642646 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:31:50.642624    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f\": container with ID starting with 17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f not found: ID does not exist" containerID="17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f"
Jul 16 18:31:50.642729 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.642658    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f"} err="failed to get container status \"17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f\": rpc error: code = NotFound desc = could not find container \"17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f\": container with ID starting with 17cde84cf705232485c528c80cd39b1e7847d565818d1fcc64b39a44354cdc8f not found: ID does not exist"
Jul 16 18:31:50.642729 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.642677    2579 scope.go:117] "RemoveContainer" containerID="61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131"
Jul 16 18:31:50.642944 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:31:50.642926    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131\": container with ID starting with 61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131 not found: ID does not exist" containerID="61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131"
Jul 16 18:31:50.643014 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.642949    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131"} err="failed to get container status \"61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131\": rpc error: code = NotFound desc = could not find container \"61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131\": container with ID starting with 61f4e77b7982f776c9e6d3ce9e64ef32c064f14a067245b3c23a766065673131 not found: ID does not exist"
Jul 16 18:31:50.647868 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.647842    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 18:31:50.655228 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:50.655195    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 16 18:31:51.102882 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:31:51.102837    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" path="/var/lib/kubelet/pods/e3295fe8-166b-4de6-8ce5-28c68eacf297/volumes"
Jul 16 18:32:12.497302 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.497261    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5c7d67c6bc-dzgk6_9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c/main/0.log"
Jul 16 18:32:12.498843 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.498825    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:32:12.592298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592179    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tls-certs\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592226    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tmp-dir\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592298 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592286    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-home\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592338    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-model-cache\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592361    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-dshm\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592402    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kserve-provision-location\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592585 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592442    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvbcz\" (UniqueName: \"kubernetes.io/projected/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kube-api-access-qvbcz\") pod \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\" (UID: \"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c\") "
Jul 16 18:32:12.592788 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592610    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:32:12.592788 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592648    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:32:12.592788 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592764    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.592788 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592782    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.592992 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.592792    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-home" (OuterVolumeSpecName: "home") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:32:12.594441 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.594413    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:32:12.594547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.594458    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kube-api-access-qvbcz" (OuterVolumeSpecName: "kube-api-access-qvbcz") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "kube-api-access-qvbcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:32:12.594706 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.594683    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-dshm" (OuterVolumeSpecName: "dshm") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:32:12.652683 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.652635    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" (UID: "9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:32:12.693547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.693509    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.693547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.693542    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.693547 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.693551    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.693773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.693559    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kserve-provision-location\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.693773 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.693569    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvbcz\" (UniqueName: \"kubernetes.io/projected/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c-kube-api-access-qvbcz\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:32:12.753484 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.753456    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_prestop-hook-test-kserve-5c7d67c6bc-dzgk6_9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c/main/0.log"
Jul 16 18:32:12.753824 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.753802    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerID="e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861" exitCode=137
Jul 16 18:32:12.753891 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.753853    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"
Jul 16 18:32:12.753947 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.753889    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" event={"ID":"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c","Type":"ContainerDied","Data":"e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861"}
Jul 16 18:32:12.753998 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.753959    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6" event={"ID":"9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c","Type":"ContainerDied","Data":"683558d9ada5ed1d86918a738d41996162dd683b05caeba9be587dabdd62115d"}
Jul 16 18:32:12.754051 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.753994    2579 scope.go:117] "RemoveContainer" containerID="e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861"
Jul 16 18:32:12.763616 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.763599    2579 scope.go:117] "RemoveContainer" containerID="27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09"
Jul 16 18:32:12.772421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.772405    2579 scope.go:117] "RemoveContainer" containerID="e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861"
Jul 16 18:32:12.772680 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:32:12.772662    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861\": container with ID starting with e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861 not found: ID does not exist" containerID="e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861"
Jul 16 18:32:12.772723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.772690    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861"} err="failed to get container status \"e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861\": rpc error: code = NotFound desc = could not find container \"e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861\": container with ID starting with e5f9d2e0dec406c5508c15eea90051658222ef7b8f9dcbd706a6027faa45c861 not found: ID does not exist"
Jul 16 18:32:12.772723 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.772707    2579 scope.go:117] "RemoveContainer" containerID="27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09"
Jul 16 18:32:12.772948 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:32:12.772930    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09\": container with ID starting with 27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09 not found: ID does not exist" containerID="27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09"
Jul 16 18:32:12.772993 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.772955    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09"} err="failed to get container status \"27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09\": rpc error: code = NotFound desc = could not find container \"27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09\": container with ID starting with 27185ee6b239a75ae4840e7002621f5b965851f7b615f47f495874a43d03ca09 not found: ID does not exist"
Jul 16 18:32:12.783461 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.783435    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"]
Jul 16 18:32:12.791206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:12.791183    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-5c7d67c6bc-dzgk6"]
Jul 16 18:32:13.102754 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:13.102725    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" path="/var/lib/kubelet/pods/9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c/volumes"
Jul 16 18:32:35.873050 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873016    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"]
Jul 16 18:32:35.873572 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873553    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerName="storage-initializer"
Jul 16 18:32:35.873651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873575    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerName="storage-initializer"
Jul 16 18:32:35.873651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873593    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerName="main"
Jul 16 18:32:35.873651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerName="main"
Jul 16 18:32:35.873651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873614    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="main"
Jul 16 18:32:35.873651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873622    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="main"
Jul 16 18:32:35.873651 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873649    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="storage-initializer"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873657    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="storage-initializer"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873667    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873697    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873706    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873715    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="storage-initializer"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873723    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="storage-initializer"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873732    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="storage-initializer"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873741    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="storage-initializer"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873818    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3295fe8-166b-4de6-8ce5-28c68eacf297" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873832    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4987bc4b-6330-4753-a140-259cb3e2af12" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873840    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c161feb-c0c5-4ff4-89cc-0e2eb9a4935c" containerName="main"
Jul 16 18:32:35.873943 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.873853    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09fe7dca-0611-4c4a-993a-920b12fff02c" containerName="main"
Jul 16 18:32:35.875921 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.875900    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.880177 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.880144    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:32:35.880177 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.880160    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-dvfqx\""
Jul 16 18:32:35.880399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.880246    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 16 18:32:35.880399 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.880300    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:32:35.880503 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.880487    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:32:35.889548 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.889522    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"]
Jul 16 18:32:35.890921 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.890893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6mzf\" (UniqueName: \"kubernetes.io/projected/1a086baa-6667-49e7-9767-bb9669cc95ac-kube-api-access-s6mzf\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.891122 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.891103    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a086baa-6667-49e7-9767-bb9669cc95ac-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.891341 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.891251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a086baa-6667-49e7-9767-bb9669cc95ac-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.992770 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.992734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a086baa-6667-49e7-9767-bb9669cc95ac-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.992948 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.992789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6mzf\" (UniqueName: \"kubernetes.io/projected/1a086baa-6667-49e7-9767-bb9669cc95ac-kube-api-access-s6mzf\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.992948 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.992831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a086baa-6667-49e7-9767-bb9669cc95ac-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.993155 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.993140    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a086baa-6667-49e7-9767-bb9669cc95ac-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:35.995181 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:35.995149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a086baa-6667-49e7-9767-bb9669cc95ac-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:36.004642 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.004615    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6mzf\" (UniqueName: \"kubernetes.io/projected/1a086baa-6667-49e7-9767-bb9669cc95ac-kube-api-access-s6mzf\") pod \"rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:36.186883 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.186787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:36.331116 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.331089    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"]
Jul 16 18:32:36.332891 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:32:36.332862    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a086baa_6667_49e7_9767_bb9669cc95ac.slice/crio-61349e23a7d0efd4d0caed47c5c764c8ca45494a49ec46cfd7d001d7492f2e8e WatchSource:0}: Error finding container 61349e23a7d0efd4d0caed47c5c764c8ca45494a49ec46cfd7d001d7492f2e8e: Status 404 returned error can't find the container with id 61349e23a7d0efd4d0caed47c5c764c8ca45494a49ec46cfd7d001d7492f2e8e
Jul 16 18:32:36.886325 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.886245    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" event={"ID":"1a086baa-6667-49e7-9767-bb9669cc95ac","Type":"ContainerStarted","Data":"634ca5f2140d611d619d8f5ba29d8fc6d20933be379d1147fa42be0553e5c3bf"}
Jul 16 18:32:36.886776 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.886343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" event={"ID":"1a086baa-6667-49e7-9767-bb9669cc95ac","Type":"ContainerStarted","Data":"61349e23a7d0efd4d0caed47c5c764c8ca45494a49ec46cfd7d001d7492f2e8e"}
Jul 16 18:32:36.886776 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.886368    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:32:36.905927 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:32:36.905879    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" podStartSLOduration=1.905864512 podStartE2EDuration="1.905864512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:32:36.905235048 +0000 UTC m=+3478.288704983" watchObservedRunningTime="2026-07-16 18:32:36.905864512 +0000 UTC m=+3478.289334436"
Jul 16 18:33:07.893718 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:07.893635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:33:46.994683 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:46.994643    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"]
Jul 16 18:33:46.996967 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:46.996950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.009031 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.009003    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"]
Jul 16 18:33:47.119851 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.119812    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.120034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.119857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjd7q\" (UniqueName: \"kubernetes.io/projected/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-kube-api-access-tjd7q\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.120034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.119923    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.120034 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.119998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.120166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.120051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.120166 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.120072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.220958 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.220921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221164 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.220979    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221164 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221164 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221026    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221164 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221164 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjd7q\" (UniqueName: \"kubernetes.io/projected/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-kube-api-access-tjd7q\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221500 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221462    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221567 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.221879 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.221855    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.223473 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.223444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.223831 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.223811    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.234515 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.234491    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjd7q\" (UniqueName: \"kubernetes.io/projected/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-kube-api-access-tjd7q\") pod \"rolling-upgrade-test-kserve-859df57bcb-hjr8m\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.308308 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.308256    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:47.449756 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:47.449730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"]
Jul 16 18:33:47.451428 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:33:47.451399    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e1f47f_4f6d_4b02_a6b4_64bf8074db79.slice/crio-a2ca73e8468fd364083e13ef4ab107eec6fdd59200394ff0ccbc9631b2c584ea WatchSource:0}: Error finding container a2ca73e8468fd364083e13ef4ab107eec6fdd59200394ff0ccbc9631b2c584ea: Status 404 returned error can't find the container with id a2ca73e8468fd364083e13ef4ab107eec6fdd59200394ff0ccbc9631b2c584ea
Jul 16 18:33:48.138647 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:48.138610    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"]
Jul 16 18:33:48.141082 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:48.141055    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"]
Jul 16 18:33:48.274873 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:48.274836    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m" event={"ID":"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79","Type":"ContainerStarted","Data":"c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4"}
Jul 16 18:33:48.275072 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:48.274900    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m" event={"ID":"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79","Type":"ContainerStarted","Data":"a2ca73e8468fd364083e13ef4ab107eec6fdd59200394ff0ccbc9631b2c584ea"}
Jul 16 18:33:48.300024 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:48.299972    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m" podStartSLOduration=2.299956216 podStartE2EDuration="2.299956216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:33:48.298309389 +0000 UTC m=+3549.681779322" watchObservedRunningTime="2026-07-16 18:33:48.299956216 +0000 UTC m=+3549.683426162"
Jul 16 18:33:48.308950 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:48.308924    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:33:48.331631 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:48.331596    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:48.331805 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:48.331665    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs podName:b4e1f47f-4f6d-4b02-a6b4-64bf8074db79 nodeName:}" failed. No retries permitted until 2026-07-16 18:33:48.831648985 +0000 UTC m=+3550.215118896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-hjr8m" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:48.836196 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:48.836161    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:48.836405 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:48.836230    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs podName:b4e1f47f-4f6d-4b02-a6b4-64bf8074db79 nodeName:}" failed. No retries permitted until 2026-07-16 18:33:49.836214589 +0000 UTC m=+3551.219684501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-hjr8m" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:49.845328 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:49.845292    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:49.845733 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:49.845366    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs podName:b4e1f47f-4f6d-4b02-a6b4-64bf8074db79 nodeName:}" failed. No retries permitted until 2026-07-16 18:33:51.845349725 +0000 UTC m=+3553.228819637 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-hjr8m" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:51.863529 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:51.863487    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:51.863968 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:51.863564    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs podName:b4e1f47f-4f6d-4b02-a6b4-64bf8074db79 nodeName:}" failed. No retries permitted until 2026-07-16 18:33:55.86354924 +0000 UTC m=+3557.247019153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-hjr8m" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:55.904190 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:55.904088    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:55.904190 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:33:55.904165    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs podName:b4e1f47f-4f6d-4b02-a6b4-64bf8074db79 nodeName:}" failed. No retries permitted until 2026-07-16 18:34:03.904149754 +0000 UTC m=+3565.287619669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-hjr8m" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:33:57.893324 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:33:57.893280    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" podUID="1a086baa-6667-49e7-9767-bb9669cc95ac" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:34:03.159485 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.159432    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" podUID="1a086baa-6667-49e7-9767-bb9669cc95ac" containerName="main" containerID="cri-o://634ca5f2140d611d619d8f5ba29d8fc6d20933be379d1147fa42be0553e5c3bf" gracePeriod=45
Jul 16 18:34:03.362650 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.362602    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a086baa-6667-49e7-9767-bb9669cc95ac" containerID="634ca5f2140d611d619d8f5ba29d8fc6d20933be379d1147fa42be0553e5c3bf" exitCode=0
Jul 16 18:34:03.362827 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.362663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" event={"ID":"1a086baa-6667-49e7-9767-bb9669cc95ac","Type":"ContainerDied","Data":"634ca5f2140d611d619d8f5ba29d8fc6d20933be379d1147fa42be0553e5c3bf"}
Jul 16 18:34:03.414646 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.414571    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:34:03.579101 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.579056    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6mzf\" (UniqueName: \"kubernetes.io/projected/1a086baa-6667-49e7-9767-bb9669cc95ac-kube-api-access-s6mzf\") pod \"1a086baa-6667-49e7-9767-bb9669cc95ac\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") "
Jul 16 18:34:03.579341 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.579115    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a086baa-6667-49e7-9767-bb9669cc95ac-tokenizer-uds\") pod \"1a086baa-6667-49e7-9767-bb9669cc95ac\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") "
Jul 16 18:34:03.579341 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.579167    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a086baa-6667-49e7-9767-bb9669cc95ac-tls-certs\") pod \"1a086baa-6667-49e7-9767-bb9669cc95ac\" (UID: \"1a086baa-6667-49e7-9767-bb9669cc95ac\") "
Jul 16 18:34:03.579463 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.579440    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a086baa-6667-49e7-9767-bb9669cc95ac-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "1a086baa-6667-49e7-9767-bb9669cc95ac" (UID: "1a086baa-6667-49e7-9767-bb9669cc95ac"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:03.581365 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.581332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a086baa-6667-49e7-9767-bb9669cc95ac-kube-api-access-s6mzf" (OuterVolumeSpecName: "kube-api-access-s6mzf") pod "1a086baa-6667-49e7-9767-bb9669cc95ac" (UID: "1a086baa-6667-49e7-9767-bb9669cc95ac"). InnerVolumeSpecName "kube-api-access-s6mzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:34:03.581495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.581377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a086baa-6667-49e7-9767-bb9669cc95ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a086baa-6667-49e7-9767-bb9669cc95ac" (UID: "1a086baa-6667-49e7-9767-bb9669cc95ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:34:03.680786 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.680686    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6mzf\" (UniqueName: \"kubernetes.io/projected/1a086baa-6667-49e7-9767-bb9669cc95ac-kube-api-access-s6mzf\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:03.680786 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.680718    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/1a086baa-6667-49e7-9767-bb9669cc95ac-tokenizer-uds\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:03.680786 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:03.680728    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a086baa-6667-49e7-9767-bb9669cc95ac-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:03.983512 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:34:03.983418    2579 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:34:03.983512 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:34:03.983495    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs podName:b4e1f47f-4f6d-4b02-a6b4-64bf8074db79 nodeName:}" failed. No retries permitted until 2026-07-16 18:34:19.983478807 +0000 UTC m=+3581.366948720 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-hjr8m" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 16 18:34:04.299401 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.299335    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m" podUID="b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" containerName="main" containerID="cri-o://c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4" gracePeriod=45
Jul 16 18:34:04.369415 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.369381    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"
Jul 16 18:34:04.369595 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.369415    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv" event={"ID":"1a086baa-6667-49e7-9767-bb9669cc95ac","Type":"ContainerDied","Data":"61349e23a7d0efd4d0caed47c5c764c8ca45494a49ec46cfd7d001d7492f2e8e"}
Jul 16 18:34:04.369595 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.369482    2579 scope.go:117] "RemoveContainer" containerID="634ca5f2140d611d619d8f5ba29d8fc6d20933be379d1147fa42be0553e5c3bf"
Jul 16 18:34:04.398738 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.398700    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"]
Jul 16 18:34:04.407294 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.407240    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-745dd558f-r9qcv"]
Jul 16 18:34:04.559194 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.559121    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:34:04.689757 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.689715    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs\") pod \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") "
Jul 16 18:34:04.689936 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.689795    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tmp-dir\") pod \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") "
Jul 16 18:34:04.689936 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.689906    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjd7q\" (UniqueName: \"kubernetes.io/projected/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-kube-api-access-tjd7q\") pod \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") "
Jul 16 18:34:04.690080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.689997    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-dshm\") pod \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") "
Jul 16 18:34:04.690080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690065    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:04.690200 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690069    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-model-cache\") pod \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") "
Jul 16 18:34:04.690200 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690128    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-home\") pod \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\" (UID: \"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79\") "
Jul 16 18:34:04.690384 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690358    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-model-cache" (OuterVolumeSpecName: "model-cache") pod "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:04.690496 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690383    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-home" (OuterVolumeSpecName: "home") pod "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:04.690617 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690598    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-model-cache\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:04.690669 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690622    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-home\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:04.690669 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.690635    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tmp-dir\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:04.692089 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.692061    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:34:04.692089 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.692069    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-dshm" (OuterVolumeSpecName: "dshm") pod "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:34:04.692228 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.692153    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-kube-api-access-tjd7q" (OuterVolumeSpecName: "kube-api-access-tjd7q") pod "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" (UID: "b4e1f47f-4f6d-4b02-a6b4-64bf8074db79"). InnerVolumeSpecName "kube-api-access-tjd7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:34:04.791725 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.791685    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:04.791725 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.791718    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjd7q\" (UniqueName: \"kubernetes.io/projected/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-kube-api-access-tjd7q\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:04.791725 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:04.791728    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79-dshm\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:05.103018 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.102979    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a086baa-6667-49e7-9767-bb9669cc95ac" path="/var/lib/kubelet/pods/1a086baa-6667-49e7-9767-bb9669cc95ac/volumes"
Jul 16 18:34:05.377848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.377759    2579 generic.go:356] "Generic (PLEG): container finished" podID="b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" containerID="c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4" exitCode=0
Jul 16 18:34:05.377848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.377818    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"
Jul 16 18:34:05.377848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.377827    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m" event={"ID":"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79","Type":"ContainerDied","Data":"c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4"}
Jul 16 18:34:05.378482 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.377894    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m" event={"ID":"b4e1f47f-4f6d-4b02-a6b4-64bf8074db79","Type":"ContainerDied","Data":"a2ca73e8468fd364083e13ef4ab107eec6fdd59200394ff0ccbc9631b2c584ea"}
Jul 16 18:34:05.378482 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.377917    2579 scope.go:117] "RemoveContainer" containerID="c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4"
Jul 16 18:34:05.386878 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.386856    2579 scope.go:117] "RemoveContainer" containerID="c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4"
Jul 16 18:34:05.387149 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:34:05.387130    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4\": container with ID starting with c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4 not found: ID does not exist" containerID="c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4"
Jul 16 18:34:05.387202 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.387158    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4"} err="failed to get container status \"c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4\": rpc error: code = NotFound desc = could not find container \"c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4\": container with ID starting with c95ba2c9cc94bcddb00b910c4e451b909016c82b7c14dc668eedef4996d0c3b4 not found: ID does not exist"
Jul 16 18:34:05.414553 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.414518    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"]
Jul 16 18:34:05.420247 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:05.420213    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-hjr8m"]
Jul 16 18:34:07.103454 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:07.103417    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" path="/var/lib/kubelet/pods/b4e1f47f-4f6d-4b02-a6b4-64bf8074db79/volumes"
Jul 16 18:34:11.709327 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709289    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"]
Jul 16 18:34:11.709751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709642    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" containerName="main"
Jul 16 18:34:11.709751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709653    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" containerName="main"
Jul 16 18:34:11.709751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709675    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a086baa-6667-49e7-9767-bb9669cc95ac" containerName="main"
Jul 16 18:34:11.709751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709681    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a086baa-6667-49e7-9767-bb9669cc95ac" containerName="main"
Jul 16 18:34:11.709751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709738    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e1f47f-4f6d-4b02-a6b4-64bf8074db79" containerName="main"
Jul 16 18:34:11.709751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.709749    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a086baa-6667-49e7-9767-bb9669cc95ac" containerName="main"
Jul 16 18:34:11.714050 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.714033    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:11.723698 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.723669    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"]
Jul 16 18:34:11.857474 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.857436    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zq4\" (UniqueName: \"kubernetes.io/projected/c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5-kube-api-access-25zq4\") pod \"llmisvc-controller-manager-7bc4499b55-c2rg5\" (UID: \"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5\") " pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:11.857668 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.857606    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5-cert\") pod \"llmisvc-controller-manager-7bc4499b55-c2rg5\" (UID: \"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5\") " pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:11.958640 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.958599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5-cert\") pod \"llmisvc-controller-manager-7bc4499b55-c2rg5\" (UID: \"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5\") " pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:11.958838 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.958661    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zq4\" (UniqueName: \"kubernetes.io/projected/c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5-kube-api-access-25zq4\") pod \"llmisvc-controller-manager-7bc4499b55-c2rg5\" (UID: \"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5\") " pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:11.961144 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.961089    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5-cert\") pod \"llmisvc-controller-manager-7bc4499b55-c2rg5\" (UID: \"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5\") " pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:11.970248 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:11.970224    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zq4\" (UniqueName: \"kubernetes.io/projected/c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5-kube-api-access-25zq4\") pod \"llmisvc-controller-manager-7bc4499b55-c2rg5\" (UID: \"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5\") " pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:12.024927 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:12.024898    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:12.161922 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:12.161899    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"]
Jul 16 18:34:12.163409 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:34:12.163378    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc9410006_cd9e_4bb1_b90c_9c5f4e4ae6e5.slice/crio-9500077e9fe7197116c84a5055de355b91776929b7c0dffc05fefc14e379a8cc WatchSource:0}: Error finding container 9500077e9fe7197116c84a5055de355b91776929b7c0dffc05fefc14e379a8cc: Status 404 returned error can't find the container with id 9500077e9fe7197116c84a5055de355b91776929b7c0dffc05fefc14e379a8cc
Jul 16 18:34:12.164754 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:12.164734    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 18:34:12.424492 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:12.424456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5" event={"ID":"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5","Type":"ContainerStarted","Data":"9500077e9fe7197116c84a5055de355b91776929b7c0dffc05fefc14e379a8cc"}
Jul 16 18:34:13.431386 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:13.431345    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5" event={"ID":"c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5","Type":"ContainerStarted","Data":"bb87d719a64b884c8e71f3dbdb8796e9ba47abde5d40acd8af8493ec5605ffa5"}
Jul 16 18:34:13.431790 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:13.431415    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:13.465539 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:13.465483    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5" podStartSLOduration=1.988026719 podStartE2EDuration="2.465467585s" totalImagesPullingTime="477.440866ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:34:13.463090789 +0000 UTC m=+3574.846560734" watchObservedRunningTime="2026-07-16 18:34:13.465467585 +0000 UTC m=+3574.848937520"
Jul 16 18:34:44.438045 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:44.438016    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-7bc4499b55-c2rg5"
Jul 16 18:34:44.496210 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:44.496179    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-d4c968599-926ss"]
Jul 16 18:34:44.496458 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:44.496392    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" podUID="be199c03-6b94-4519-a3d3-bef344f4115d" containerName="manager" containerID="cri-o://d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667" gracePeriod=30
Jul 16 18:34:48.546996 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:48.546969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:34:48.554862 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:48.554839    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:34:52.115665 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.115624    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" podUID="be199c03-6b94-4519-a3d3-bef344f4115d" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.31:8081/readyz\": dial tcp 10.133.0.31:8081: connect: connection refused"
Jul 16 18:34:52.244751 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.244727    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 18:34:52.309519 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.309489    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be199c03-6b94-4519-a3d3-bef344f4115d-cert\") pod \"be199c03-6b94-4519-a3d3-bef344f4115d\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") "
Jul 16 18:34:52.309709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.309558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kbzz\" (UniqueName: \"kubernetes.io/projected/be199c03-6b94-4519-a3d3-bef344f4115d-kube-api-access-7kbzz\") pod \"be199c03-6b94-4519-a3d3-bef344f4115d\" (UID: \"be199c03-6b94-4519-a3d3-bef344f4115d\") "
Jul 16 18:34:52.311583 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.311562    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be199c03-6b94-4519-a3d3-bef344f4115d-cert" (OuterVolumeSpecName: "cert") pod "be199c03-6b94-4519-a3d3-bef344f4115d" (UID: "be199c03-6b94-4519-a3d3-bef344f4115d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:34:52.311742 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.311718    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be199c03-6b94-4519-a3d3-bef344f4115d-kube-api-access-7kbzz" (OuterVolumeSpecName: "kube-api-access-7kbzz") pod "be199c03-6b94-4519-a3d3-bef344f4115d" (UID: "be199c03-6b94-4519-a3d3-bef344f4115d"). InnerVolumeSpecName "kube-api-access-7kbzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:34:52.410131 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.410095    2579 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be199c03-6b94-4519-a3d3-bef344f4115d-cert\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:52.410131 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.410124    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kbzz\" (UniqueName: \"kubernetes.io/projected/be199c03-6b94-4519-a3d3-bef344f4115d-kube-api-access-7kbzz\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:34:52.639800 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.639713    2579 generic.go:356] "Generic (PLEG): container finished" podID="be199c03-6b94-4519-a3d3-bef344f4115d" containerID="d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667" exitCode=0
Jul 16 18:34:52.639800 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.639762    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-d4c968599-926ss"
Jul 16 18:34:52.639991 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.639797    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" event={"ID":"be199c03-6b94-4519-a3d3-bef344f4115d","Type":"ContainerDied","Data":"d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667"}
Jul 16 18:34:52.639991 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.639858    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-d4c968599-926ss" event={"ID":"be199c03-6b94-4519-a3d3-bef344f4115d","Type":"ContainerDied","Data":"2f3443c9584410964270def6000720ffd09429f8397a0fee1680d02e06c76570"}
Jul 16 18:34:52.639991 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.639885    2579 scope.go:117] "RemoveContainer" containerID="d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667"
Jul 16 18:34:52.650633 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.650614    2579 scope.go:117] "RemoveContainer" containerID="d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667"
Jul 16 18:34:52.650863 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:34:52.650844    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667\": container with ID starting with d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667 not found: ID does not exist" containerID="d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667"
Jul 16 18:34:52.650940 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.650871    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667"} err="failed to get container status \"d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667\": rpc error: code = NotFound desc = could not find container \"d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667\": container with ID starting with d82fa699b837789720d56701a3cbdf1567f15697be82cc609b0810d21041e667 not found: ID does not exist"
Jul 16 18:34:52.665763 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.665733    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/llmisvc-controller-manager-d4c968599-926ss"]
Jul 16 18:34:52.671159 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:52.671135    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/llmisvc-controller-manager-d4c968599-926ss"]
Jul 16 18:34:53.103017 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:34:53.102982    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be199c03-6b94-4519-a3d3-bef344f4115d" path="/var/lib/kubelet/pods/be199c03-6b94-4519-a3d3-bef344f4115d/volumes"
Jul 16 18:36:02.551806 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.551715    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"]
Jul 16 18:36:02.552235 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.552105    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be199c03-6b94-4519-a3d3-bef344f4115d" containerName="manager"
Jul 16 18:36:02.552235 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.552119    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="be199c03-6b94-4519-a3d3-bef344f4115d" containerName="manager"
Jul 16 18:36:02.552235 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.552189    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="be199c03-6b94-4519-a3d3-bef344f4115d" containerName="manager"
Jul 16 18:36:02.555059 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.555038    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.559629 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.559601    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:36:02.559774 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.559603    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:36:02.559774 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.559614    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 16 18:36:02.559774 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.559688    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-gjtst\""
Jul 16 18:36:02.559774 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.559702    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:36:02.567901 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.567876    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"]
Jul 16 18:36:02.716978 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.716943    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.717165 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.716989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.717165 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.717080    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm88s\" (UniqueName: \"kubernetes.io/projected/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-kube-api-access-fm88s\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.818633 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.818542    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm88s\" (UniqueName: \"kubernetes.io/projected/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-kube-api-access-fm88s\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.818633 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.818627    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.818888 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.818658    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.819016 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.818996    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.821069 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.821043    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.828972 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.828942    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm88s\" (UniqueName: \"kubernetes.io/projected/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-kube-api-access-fm88s\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:02.864693 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:02.864667    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:03.003379 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:03.003351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"]
Jul 16 18:36:03.005321 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:36:03.005288    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6642ddbe_e965_4b3d_b4f5_617ef60bcb26.slice/crio-6eaefa9a72830370d0863d0da7acf20684f85eea71dd47ad27c0c7ef27e1e3ba WatchSource:0}: Error finding container 6eaefa9a72830370d0863d0da7acf20684f85eea71dd47ad27c0c7ef27e1e3ba: Status 404 returned error can't find the container with id 6eaefa9a72830370d0863d0da7acf20684f85eea71dd47ad27c0c7ef27e1e3ba
Jul 16 18:36:04.015902 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:04.015851    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" event={"ID":"6642ddbe-e965-4b3d-b4f5-617ef60bcb26","Type":"ContainerStarted","Data":"93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f"}
Jul 16 18:36:04.016338 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:04.015945    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" event={"ID":"6642ddbe-e965-4b3d-b4f5-617ef60bcb26","Type":"ContainerStarted","Data":"6eaefa9a72830370d0863d0da7acf20684f85eea71dd47ad27c0c7ef27e1e3ba"}
Jul 16 18:36:04.016338 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:04.015974    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:36:04.041976 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:04.041927    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podStartSLOduration=2.041914725 podStartE2EDuration="2.041914725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:36:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:36:04.039002203 +0000 UTC m=+3685.422472135" watchObservedRunningTime="2026-07-16 18:36:04.041914725 +0000 UTC m=+3685.425384658"
Jul 16 18:36:35.024663 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:35.024622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:36:45.024546 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:45.024506    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:36:55.024322 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:36:55.024261    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:37:05.024914 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:05.024874    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:37:05.348503 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:05.348468    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"]
Jul 16 18:37:05.350373 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:05.350337    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:37:15.350153 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:15.350114    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:37:20.369421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.369376    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main" containerID="cri-o://93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f" gracePeriod=45
Jul 16 18:37:20.618248 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.618223    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:37:20.699619 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.699533    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tokenizer-uds\") pod \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") "
Jul 16 18:37:20.699619 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.699574    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm88s\" (UniqueName: \"kubernetes.io/projected/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-kube-api-access-fm88s\") pod \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") "
Jul 16 18:37:20.699813 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.699637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tls-certs\") pod \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\" (UID: \"6642ddbe-e965-4b3d-b4f5-617ef60bcb26\") "
Jul 16 18:37:20.699866 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.699835    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "6642ddbe-e965-4b3d-b4f5-617ef60bcb26" (UID: "6642ddbe-e965-4b3d-b4f5-617ef60bcb26"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:37:20.701641 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.701615    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6642ddbe-e965-4b3d-b4f5-617ef60bcb26" (UID: "6642ddbe-e965-4b3d-b4f5-617ef60bcb26"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:37:20.701749 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.701674    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-kube-api-access-fm88s" (OuterVolumeSpecName: "kube-api-access-fm88s") pod "6642ddbe-e965-4b3d-b4f5-617ef60bcb26" (UID: "6642ddbe-e965-4b3d-b4f5-617ef60bcb26"). InnerVolumeSpecName "kube-api-access-fm88s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:37:20.800201 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.800163    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:37:20.800201 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.800198    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-tokenizer-uds\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:37:20.800201 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:20.800207    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm88s\" (UniqueName: \"kubernetes.io/projected/6642ddbe-e965-4b3d-b4f5-617ef60bcb26-kube-api-access-fm88s\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:37:21.436500 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.436410    2579 generic.go:356] "Generic (PLEG): container finished" podID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerID="93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f" exitCode=0
Jul 16 18:37:21.436500 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.436453    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"
Jul 16 18:37:21.436996 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.436496    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" event={"ID":"6642ddbe-e965-4b3d-b4f5-617ef60bcb26","Type":"ContainerDied","Data":"93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f"}
Jul 16 18:37:21.436996 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.436551    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j" event={"ID":"6642ddbe-e965-4b3d-b4f5-617ef60bcb26","Type":"ContainerDied","Data":"6eaefa9a72830370d0863d0da7acf20684f85eea71dd47ad27c0c7ef27e1e3ba"}
Jul 16 18:37:21.436996 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.436578    2579 scope.go:117] "RemoveContainer" containerID="93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f"
Jul 16 18:37:21.445718 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.445699    2579 scope.go:117] "RemoveContainer" containerID="93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f"
Jul 16 18:37:21.445989 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:37:21.445975    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f\": container with ID starting with 93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f not found: ID does not exist" containerID="93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f"
Jul 16 18:37:21.446050 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.445996    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f"} err="failed to get container status \"93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f\": rpc error: code = NotFound desc = could not find container \"93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f\": container with ID starting with 93c13cbf5b51d75ebab7b3e122a3e4211f4f82505a162af8b317419bc065670f not found: ID does not exist"
Jul 16 18:37:21.458791 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.458766    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"]
Jul 16 18:37:21.464092 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:21.464069    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-85cd85fj5r7j"]
Jul 16 18:37:23.109600 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.109563    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" path="/var/lib/kubelet/pods/6642ddbe-e965-4b3d-b4f5-617ef60bcb26/volumes"
Jul 16 18:37:23.557854 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.557820    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"]
Jul 16 18:37:23.558236 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.558222    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main"
Jul 16 18:37:23.558320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.558238    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main"
Jul 16 18:37:23.558320 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.558313    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6642ddbe-e965-4b3d-b4f5-617ef60bcb26" containerName="main"
Jul 16 18:37:23.563410 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.563373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.566517 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.566487    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-j4fsz\""
Jul 16 18:37:23.566661 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.566520    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-vczbt\""
Jul 16 18:37:23.567728 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.567705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:37:23.567844 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.567705    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:37:23.568003 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.567990    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 16 18:37:23.573848 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.573820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"]
Jul 16 18:37:23.625340 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.625304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-home\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.625540 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.625379    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.625540 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.625401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.625540 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.625442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.625540 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.625527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22a527d-b525-419d-91ba-a9a2f6b39dcc-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.625765 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.625588    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbh8h\" (UniqueName: \"kubernetes.io/projected/a22a527d-b525-419d-91ba-a9a2f6b39dcc-kube-api-access-rbh8h\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.726780 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.726749    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.726984 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.726793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22a527d-b525-419d-91ba-a9a2f6b39dcc-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.726984 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.726831    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbh8h\" (UniqueName: \"kubernetes.io/projected/a22a527d-b525-419d-91ba-a9a2f6b39dcc-kube-api-access-rbh8h\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.726984 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.726860    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-home\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.726984 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.726924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.726984 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.726947    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.727384 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.727352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-home\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.727384 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.727378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.727562 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.727395    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.729154 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.729133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a22a527d-b525-419d-91ba-a9a2f6b39dcc-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.729343 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.729327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a22a527d-b525-419d-91ba-a9a2f6b39dcc-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.737165 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.737142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbh8h\" (UniqueName: \"kubernetes.io/projected/a22a527d-b525-419d-91ba-a9a2f6b39dcc-kube-api-access-rbh8h\") pod \"scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24\" (UID: \"a22a527d-b525-419d-91ba-a9a2f6b39dcc\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:23.874433 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:23.874355    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:24.055460 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:37:24.055424    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22a527d_b525_419d_91ba_a9a2f6b39dcc.slice/crio-7cb9abb7b25413c1ec01def18b7c4f2cdead2a98d4f56af0c033dbcfa4a13411 WatchSource:0}: Error finding container 7cb9abb7b25413c1ec01def18b7c4f2cdead2a98d4f56af0c033dbcfa4a13411: Status 404 returned error can't find the container with id 7cb9abb7b25413c1ec01def18b7c4f2cdead2a98d4f56af0c033dbcfa4a13411
Jul 16 18:37:24.063421 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:24.063395    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"]
Jul 16 18:37:24.456668 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:24.456630    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24" event={"ID":"a22a527d-b525-419d-91ba-a9a2f6b39dcc","Type":"ContainerStarted","Data":"ef8369b9a959790d1ff30ed39cb05bdc6fccc8da125e8157b08a37ace6eeb39f"}
Jul 16 18:37:24.457206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:24.456690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24" event={"ID":"a22a527d-b525-419d-91ba-a9a2f6b39dcc","Type":"ContainerStarted","Data":"7cb9abb7b25413c1ec01def18b7c4f2cdead2a98d4f56af0c033dbcfa4a13411"}
Jul 16 18:37:24.457206 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:24.456724    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:25.463753 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:25.463713    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24" event={"ID":"a22a527d-b525-419d-91ba-a9a2f6b39dcc","Type":"ContainerStarted","Data":"2d6e112fed67c3e7fcab4e4a56237ec2d2f0a4f9370016a7eba78f54debfd345"}
Jul 16 18:37:25.487919 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:25.487852    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24" podStartSLOduration=2.487833499 podStartE2EDuration="2.487833499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:37:25.484822739 +0000 UTC m=+3766.868292673" watchObservedRunningTime="2026-07-16 18:37:25.487833499 +0000 UTC m=+3766.871303438"
Jul 16 18:37:25.874492 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:25.874455    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:33.875396 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:33.875299    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:33.888517 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:33.888487    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:34.520807 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:34.520773    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:34.521999 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:34.521982    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24"
Jul 16 18:37:54.876494 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.876461    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"]
Jul 16 18:37:54.880030 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.880013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:54.885215 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.885190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-jdlf9\""
Jul 16 18:37:54.885377 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.885195    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 16 18:37:54.891029 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.891000    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"]
Jul 16 18:37:54.905153 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.905128    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:54.905306 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.905181    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdbb\" (UniqueName: \"kubernetes.io/projected/ec9d9914-6968-4e3d-b6e6-f26a1236e978-kube-api-access-4vdbb\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:54.905306 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:54.905220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.006701 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.006664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdbb\" (UniqueName: \"kubernetes.io/projected/ec9d9914-6968-4e3d-b6e6-f26a1236e978-kube-api-access-4vdbb\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.006883 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.006711    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.006883 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.006762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.007115 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.007098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tokenizer-uds\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.009437 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.009406    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.016525 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.016497    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdbb\" (UniqueName: \"kubernetes.io/projected/ec9d9914-6968-4e3d-b6e6-f26a1236e978-kube-api-access-4vdbb\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.189120 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.189034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.325910 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.325871    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"]
Jul 16 18:37:55.328483 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:37:55.328451    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9d9914_6968_4e3d_b6e6_f26a1236e978.slice/crio-f0144932a9e1aebb914828741fa4b58f957c27b3d5259149a34cf9b58fa4cf02 WatchSource:0}: Error finding container f0144932a9e1aebb914828741fa4b58f957c27b3d5259149a34cf9b58fa4cf02: Status 404 returned error can't find the container with id f0144932a9e1aebb914828741fa4b58f957c27b3d5259149a34cf9b58fa4cf02
Jul 16 18:37:55.630802 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.630764    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" event={"ID":"ec9d9914-6968-4e3d-b6e6-f26a1236e978","Type":"ContainerStarted","Data":"67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e"}
Jul 16 18:37:55.630976 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.630821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" event={"ID":"ec9d9914-6968-4e3d-b6e6-f26a1236e978","Type":"ContainerStarted","Data":"f0144932a9e1aebb914828741fa4b58f957c27b3d5259149a34cf9b58fa4cf02"}
Jul 16 18:37:55.630976 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.630933    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:37:55.664496 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:37:55.664438    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" podStartSLOduration=1.664422633 podStartE2EDuration="1.664422633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:37:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:37:55.660855116 +0000 UTC m=+3797.044325053" watchObservedRunningTime="2026-07-16 18:37:55.664422633 +0000 UTC m=+3797.047892566"
Jul 16 18:38:26.637529 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:26.637499    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:38:28.276454 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:28.276422    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"]
Jul 16 18:38:36.637662 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:36.637622    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" podUID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 16 18:38:43.297818 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.297776    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" podUID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" containerName="main" containerID="cri-o://67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e" gracePeriod=45
Jul 16 18:38:43.551196 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.551130    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:38:43.726709 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.726674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vdbb\" (UniqueName: \"kubernetes.io/projected/ec9d9914-6968-4e3d-b6e6-f26a1236e978-kube-api-access-4vdbb\") pod \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") "
Jul 16 18:38:43.726903 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.726737    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tls-certs\") pod \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") "
Jul 16 18:38:43.726903 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.726891    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tokenizer-uds\") pod \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\" (UID: \"ec9d9914-6968-4e3d-b6e6-f26a1236e978\") "
Jul 16 18:38:43.727184 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.727156    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "ec9d9914-6968-4e3d-b6e6-f26a1236e978" (UID: "ec9d9914-6968-4e3d-b6e6-f26a1236e978"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:38:43.728847 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.728825    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9d9914-6968-4e3d-b6e6-f26a1236e978-kube-api-access-4vdbb" (OuterVolumeSpecName: "kube-api-access-4vdbb") pod "ec9d9914-6968-4e3d-b6e6-f26a1236e978" (UID: "ec9d9914-6968-4e3d-b6e6-f26a1236e978"). InnerVolumeSpecName "kube-api-access-4vdbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:38:43.728939 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.728893    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec9d9914-6968-4e3d-b6e6-f26a1236e978" (UID: "ec9d9914-6968-4e3d-b6e6-f26a1236e978"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:38:43.828385 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.828301    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tokenizer-uds\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.828385 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.828333    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vdbb\" (UniqueName: \"kubernetes.io/projected/ec9d9914-6968-4e3d-b6e6-f26a1236e978-kube-api-access-4vdbb\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.828385 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.828344    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec9d9914-6968-4e3d-b6e6-f26a1236e978-tls-certs\") on node \"ip-10-0-132-31.ec2.internal\" DevicePath \"\""
Jul 16 18:38:43.894318 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.894285    2579 generic.go:356] "Generic (PLEG): container finished" podID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" containerID="67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e" exitCode=0
Jul 16 18:38:43.894318 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.894321    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"
Jul 16 18:38:43.894571 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.894355    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" event={"ID":"ec9d9914-6968-4e3d-b6e6-f26a1236e978","Type":"ContainerDied","Data":"67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e"}
Jul 16 18:38:43.894571 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.894401    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5" event={"ID":"ec9d9914-6968-4e3d-b6e6-f26a1236e978","Type":"ContainerDied","Data":"f0144932a9e1aebb914828741fa4b58f957c27b3d5259149a34cf9b58fa4cf02"}
Jul 16 18:38:43.894571 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.894421    2579 scope.go:117] "RemoveContainer" containerID="67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e"
Jul 16 18:38:43.904286 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.904255    2579 scope.go:117] "RemoveContainer" containerID="67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e"
Jul 16 18:38:43.904553 ip-10-0-132-31 kubenswrapper[2579]: E0716 18:38:43.904535    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e\": container with ID starting with 67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e not found: ID does not exist" containerID="67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e"
Jul 16 18:38:43.904607 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.904563    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e"} err="failed to get container status \"67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e\": rpc error: code = NotFound desc = could not find container \"67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e\": container with ID starting with 67974876abc81e1e40d576d2de8f8e05e032d7547950da651935a9988d2dc60e not found: ID does not exist"
Jul 16 18:38:43.918817 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.918792    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"]
Jul 16 18:38:43.923968 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:43.923947    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-router-schedxrdl5"]
Jul 16 18:38:45.103682 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:38:45.103646    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" path="/var/lib/kubelet/pods/ec9d9914-6968-4e3d-b6e6-f26a1236e978/volumes"
Jul 16 18:39:48.583874 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:39:48.583844    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:39:48.593823 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:39:48.593802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:44:48.622031 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:44:48.621921    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:44:48.632615 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:44:48.632592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:49:48.657876 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:49:48.657759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:49:48.670767 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:49:48.670741    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:54:48.695356 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:54:48.695228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:54:48.708261 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:54:48.708242    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vm2hw_7c1955ab-f9aa-431c-bbcf-f234b5f25a0f/ovn-acl-logging/0.log"
Jul 16 18:56:23.583577 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:23.583544    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:23.593472 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:23.593445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:42.596310 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:42.596283    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:42.673838 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:42.673810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:42.688423 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:42.688395    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:44.248706 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:44.248677    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:44.306937 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:44.306911    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:44.320707 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:44.320678    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:45.841501 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:45.841457    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:45.896203 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:45.896167    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:45.910359 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:45.910328    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:47.458400 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:47.458371    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:47.527905 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:47.527870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:47.541694 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:47.541668    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:49.085694 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:49.085659    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:49.151954 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:49.151925    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:49.171029 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:49.171002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:50.714137 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:50.714111    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:50.769894 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:50.769867    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:50.783947 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:50.783927    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:52.314213 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:52.314184    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:52.392055 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:52.392024    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:52.412093 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:52.412067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:53.976628 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:53.976596    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:54.034769 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:54.034743    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:54.052316 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:54.052286    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:55.896477 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:55.896452    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:55.957558 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:55.957533    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:55.972927 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:55.972888    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:57.530781 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:57.530756    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:57.590640 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:57.590608    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:57.605813 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:57.605786    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:56:59.138786 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:59.138759    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:56:59.192126 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:59.192098    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:56:59.206123 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:56:59.206100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:57:00.730117 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:00.730088    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:57:00.789195 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:00.789161    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:57:00.811067 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:00.811009    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:57:02.361144 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:02.361115    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:57:02.416180 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:02.416150    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:57:02.433180 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:02.433154    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:57:03.977594 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:03.977557    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:57:04.032625 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:04.032598    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:57:04.047996 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:04.047975    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:57:05.604853 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:05.604824    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:57:05.662010 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:05.661984    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:57:05.676801 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:05.676770    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:57:07.203209 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:07.203182    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_e2e-pvc-model-download-v8gm9_9a9a2926-049a-43b6-b6bc-41fcfe89713a/storage-initializer/0.log"
Jul 16 18:57:07.260993 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:07.260963    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/main/0.log"
Jul 16 18:57:07.276161 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:07.276134    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-74c855b64f-5zk24_a22a527d-b525-419d-91ba-a9a2f6b39dcc/llm-d-routing-sidecar/0.log"
Jul 16 18:57:09.204613 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:09.204576    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:09.222153 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:09.222127    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:09.235229 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:09.235203    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:10.531372 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:10.531343    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:10.548811 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:10.548783    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:10.562503 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:10.562476    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:11.835168 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:11.835141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:11.853325 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:11.853297    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:11.868923 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:11.868895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:13.133056 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:13.133021    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:13.151196 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:13.151163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:13.164424 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:13.164402    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:14.207929 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:14.207898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:15.502054 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:15.502024    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:16.812380 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:16.812350    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:18.055473 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:18.055445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:19.305941 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:19.305913    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:20.596711 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:20.596675    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:21.866001 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:21.865971    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:23.137035 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:23.137004    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:24.397737 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:24.397704    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:25.651145 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:25.651119    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:26.920632 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:26.920604    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:32.653876 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:32.653846    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rngnz_ec9208e8-03c0-4bf2-85f6-7e1470d0da5f/global-pull-secret-syncer/0.log"
Jul 16 18:57:32.761435 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:32.761407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-4rtn2_d05dd297-4cd0-4616-8cc1-53d85de33013/konnectivity-agent/0.log"
Jul 16 18:57:32.864801 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:32.864767    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-31.ec2.internal_56b83bb518a676ee4805f90256f69199/haproxy/0.log"
Jul 16 18:57:37.008020 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:37.007992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-85fd4b8b9d-htdmf_4b53f64d-35e4-4de9-98c6-9ad9f369b8bc/kuadrant-console-plugin/0.log"
Jul 16 18:57:38.725535 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.725498    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/alertmanager/0.log"
Jul 16 18:57:38.756360 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.756326    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/config-reloader/0.log"
Jul 16 18:57:38.786887 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.786862    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/kube-rbac-proxy-web/0.log"
Jul 16 18:57:38.812286 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.812243    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/kube-rbac-proxy/0.log"
Jul 16 18:57:38.843458 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.843435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/kube-rbac-proxy-metric/0.log"
Jul 16 18:57:38.867441 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.867415    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/prom-label-proxy/0.log"
Jul 16 18:57:38.891019 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.890995    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5fd6b012-1335-4277-9bf6-36cbeab4d496/init-config-reloader/0.log"
Jul 16 18:57:38.938810 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:38.938784    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-gfs8k_ff729380-dbc7-4143-863d-f25fdc3730aa/cluster-monitoring-operator/0.log"
Jul 16 18:57:39.087708 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.087683    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-pwbl4_2a847717-34da-4618-809f-2d42178cf482/monitoring-plugin/0.log"
Jul 16 18:57:39.121321 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.121296    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j6rrx_d9dbc514-88f7-4953-bd27-f3c04cd4d3c0/node-exporter/0.log"
Jul 16 18:57:39.145936 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.145907    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j6rrx_d9dbc514-88f7-4953-bd27-f3c04cd4d3c0/kube-rbac-proxy/0.log"
Jul 16 18:57:39.169791 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.169764    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-j6rrx_d9dbc514-88f7-4953-bd27-f3c04cd4d3c0/init-textfile/0.log"
Jul 16 18:57:39.715084 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.715056    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-84sw4_a219f952-93d0-4d26-92f1-5714b80d7390/prometheus-operator-admission-webhook/0.log"
Jul 16 18:57:39.745108 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.745084    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-575fc68c48-9bwv8_a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0/telemeter-client/0.log"
Jul 16 18:57:39.769684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.769658    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-575fc68c48-9bwv8_a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0/reload/0.log"
Jul 16 18:57:39.793446 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.793420    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-575fc68c48-9bwv8_a3eacfa3-2c8a-4b9f-8297-0f141a2e7dd0/kube-rbac-proxy/0.log"
Jul 16 18:57:39.832722 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.832688    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f9d78dc5c-hhzqd_a330282b-6e6b-4dfa-8b04-f60dcae027e2/thanos-query/0.log"
Jul 16 18:57:39.864439 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.864411    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f9d78dc5c-hhzqd_a330282b-6e6b-4dfa-8b04-f60dcae027e2/kube-rbac-proxy-web/0.log"
Jul 16 18:57:39.882219 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.882191    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f9d78dc5c-hhzqd_a330282b-6e6b-4dfa-8b04-f60dcae027e2/kube-rbac-proxy/0.log"
Jul 16 18:57:39.906687 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.906660    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f9d78dc5c-hhzqd_a330282b-6e6b-4dfa-8b04-f60dcae027e2/prom-label-proxy/0.log"
Jul 16 18:57:39.930347 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.930325    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f9d78dc5c-hhzqd_a330282b-6e6b-4dfa-8b04-f60dcae027e2/kube-rbac-proxy-rules/0.log"
Jul 16 18:57:39.956028 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:39.956002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7f9d78dc5c-hhzqd_a330282b-6e6b-4dfa-8b04-f60dcae027e2/kube-rbac-proxy-metrics/0.log"
Jul 16 18:57:42.506499 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.506469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6466cc997d-hbh2z_817965e3-c77c-4ee2-89da-170b65ce8276/console/0.log"
Jul 16 18:57:42.786774 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.786680    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"]
Jul 16 18:57:42.787186 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.787161    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" containerName="main"
Jul 16 18:57:42.787186 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.787186    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" containerName="main"
Jul 16 18:57:42.787411 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.787261    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9d9914-6968-4e3d-b6e6-f26a1236e978" containerName="main"
Jul 16 18:57:42.790495 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.790475    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.793453 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.793431    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt8tj\"/\"openshift-service-ca.crt\""
Jul 16 18:57:42.794677 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.794659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kt8tj\"/\"default-dockercfg-px8tg\""
Jul 16 18:57:42.794748 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.794665    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kt8tj\"/\"kube-root-ca.crt\""
Jul 16 18:57:42.800635 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.800614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"]
Jul 16 18:57:42.804576 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.804554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-lib-modules\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.804675 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.804595    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-podres\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.804675 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.804629    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4kpj\" (UniqueName: \"kubernetes.io/projected/3717ab10-63cd-4264-b066-195c09142a5b-kube-api-access-r4kpj\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.804675 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.804665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-sys\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.804805 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.804694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-proc\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.905917 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.905881    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-sys\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.905927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-proc\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.905970    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-lib-modules\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.906000    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-podres\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.906013    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-sys\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.906022    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4kpj\" (UniqueName: \"kubernetes.io/projected/3717ab10-63cd-4264-b066-195c09142a5b-kube-api-access-r4kpj\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906112 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.906068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-proc\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906408 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.906149    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-podres\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.906408 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.906172    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3717ab10-63cd-4264-b066-195c09142a5b-lib-modules\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:42.915466 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:42.915449    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4kpj\" (UniqueName: \"kubernetes.io/projected/3717ab10-63cd-4264-b066-195c09142a5b-kube-api-access-r4kpj\") pod \"perf-node-gather-daemonset-qv956\" (UID: \"3717ab10-63cd-4264-b066-195c09142a5b\") " pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:43.100907 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.100875    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:43.163080 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.162992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-2jz9x_25a1915f-e815-4264-a775-dd8cf4709da4/volume-data-source-validator/0.log"
Jul 16 18:57:43.236431 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.236405    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"]
Jul 16 18:57:43.237522 ip-10-0-132-31 kubenswrapper[2579]: W0716 18:57:43.237488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3717ab10_63cd_4264_b066_195c09142a5b.slice/crio-ffe4452294b00f0b1908bb54aeea62f69e3f2ae9cb2901d6ed88c0a5d833e36d WatchSource:0}: Error finding container ffe4452294b00f0b1908bb54aeea62f69e3f2ae9cb2901d6ed88c0a5d833e36d: Status 404 returned error can't find the container with id ffe4452294b00f0b1908bb54aeea62f69e3f2ae9cb2901d6ed88c0a5d833e36d
Jul 16 18:57:43.970629 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.970586    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956" event={"ID":"3717ab10-63cd-4264-b066-195c09142a5b","Type":"ContainerStarted","Data":"edf589f033ebfcdc7c0ac09ced213a0eb3c0c2de69a95d0710499c69823129d3"}
Jul 16 18:57:43.971062 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.970642    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956" event={"ID":"3717ab10-63cd-4264-b066-195c09142a5b","Type":"ContainerStarted","Data":"ffe4452294b00f0b1908bb54aeea62f69e3f2ae9cb2901d6ed88c0a5d833e36d"}
Jul 16 18:57:43.971062 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.970715    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:43.989628 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:43.989575    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956" podStartSLOduration=1.989559955 podStartE2EDuration="1.989559955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:57:43.988456829 +0000 UTC m=+4985.371926764" watchObservedRunningTime="2026-07-16 18:57:43.989559955 +0000 UTC m=+4985.373029888"
Jul 16 18:57:44.303846 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:44.303820    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qtjkf_ec720fcf-f94b-433d-907b-a361b69fa425/dns/0.log"
Jul 16 18:57:44.334223 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:44.334189    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qtjkf_ec720fcf-f94b-433d-907b-a361b69fa425/kube-rbac-proxy/0.log"
Jul 16 18:57:44.355006 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:44.354980    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-254d8_1bcb2707-bb60-4d1d-b477-bbe7cc04c710/dns-node-resolver/0.log"
Jul 16 18:57:45.084068 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:45.084042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wxrdp_71f18ce5-721a-48b2-a9d9-3aa00b1ff7a1/node-ca/0.log"
Jul 16 18:57:46.458099 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:46.458067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:46.476775 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:46.476744    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:46.500505 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:46.500474    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:47.016037 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:47.016006    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5blf5_0a5b3a7e-65e1-4ef1-ab25-9944805aac4f/serve-healthcheck-canary/0.log"
Jul 16 18:57:47.605792 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:47.605762    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fsbjf_2090a0cd-36b7-43d1-87b8-ed79bb02df11/kube-rbac-proxy/0.log"
Jul 16 18:57:47.643684 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:47.643656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fsbjf_2090a0cd-36b7-43d1-87b8-ed79bb02df11/exporter/0.log"
Jul 16 18:57:47.681906 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:47.681878    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fsbjf_2090a0cd-36b7-43d1-87b8-ed79bb02df11/extractor/0.log"
Jul 16 18:57:49.986457 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:49.986428    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kt8tj/perf-node-gather-daemonset-qv956"
Jul 16 18:57:50.589019 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:50.588991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c87b7c6dd-v85f7_57ad18f4-aef4-4593-9809-a6d2134f458f/manager/0.log"
Jul 16 18:57:51.326489 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:51.326452    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-7bc4499b55-c2rg5_c9410006-cd9e-4bb1-b90c-9c5f4e4ae6e5/manager/0.log"
Jul 16 18:57:52.295113 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:52.295086    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4sjxq_34e08602-8875-4c94-9f8b-07a239066a8e/s3-init/0.log"
Jul 16 18:57:52.313994 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:52.313969    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-4sjxq_34e08602-8875-4c94-9f8b-07a239066a8e/download-hf-model/0.log"
Jul 16 18:57:53.423322 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:53.423291    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:53.443470 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:53.443436    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:53.455770 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:53.455740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:54.873669 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:54.873637    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/0.log"
Jul 16 18:57:54.894896 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:54.894808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-default-866675cbb8-qnxtj_67837974-612e-4f3a-9fc1-e70b10efec4e/istio-proxy/1.log"
Jul 16 18:57:54.909091 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:54.909065    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7454758546-dtbkw_ec5697a2-8c24-47ed-91e6-d3c2e7827bd9/router/0.log"
Jul 16 18:57:59.069450 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.069419    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/kube-multus-additional-cni-plugins/0.log"
Jul 16 18:57:59.092016 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.091984    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/egress-router-binary-copy/0.log"
Jul 16 18:57:59.113769 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.113740    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/cni-plugins/0.log"
Jul 16 18:57:59.135909 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.135857    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/bond-cni-plugin/0.log"
Jul 16 18:57:59.158636 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.158614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/routeoverride-cni/0.log"
Jul 16 18:57:59.181628 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.181602    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/whereabouts-cni-bincopy/0.log"
Jul 16 18:57:59.204406 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.204377    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pgzsj_813304a6-08e0-48ab-a660-f8acfe893c99/whereabouts-cni/0.log"
Jul 16 18:57:59.276441 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.276414    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l8rvw_441e47ae-e5d0-4faf-ae89-d8f798600e46/kube-multus/0.log"
Jul 16 18:57:59.392550 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.392469    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d5x6f_b72558f3-3a3d-480a-9540-c427f35bfc10/network-metrics-daemon/0.log"
Jul 16 18:57:59.415152 ip-10-0-132-31 kubenswrapper[2579]: I0716 18:57:59.415124    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-d5x6f_b72558f3-3a3d-480a-9540-c427f35bfc10/kube-rbac-proxy/0.log"