Jul 27 17:01:31.317797 ip-10-0-132-194 systemd[1]: Starting Kubernetes Kubelet...
Jul 27 17:01:31.645515 ip-10-0-132-194 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:01:31.645515 ip-10-0-132-194 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 27 17:01:31.645515 ip-10-0-132-194 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:01:31.645515 ip-10-0-132-194 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 27 17:01:31.645515 ip-10-0-132-194 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 27 17:01:31.646245 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.645565    2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 27 17:01:31.649457 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649442    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:01:31.649457 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649457    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649461    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649465    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649468    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649484    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649487    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649490    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649493    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649496    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649498    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649501    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649504    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649506    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649509    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649512    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649515    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649517    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649519    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649529    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649532    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:01:31.649539 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649535    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649537    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649540    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649542    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649545    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649548    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649550    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649553    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649555    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649558    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649561    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649563    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649566    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649570    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649574    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649577    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649579    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649582    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649584    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:01:31.650022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649587    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649590    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649592    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649594    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649597    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649599    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649602    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649605    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649607    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649609    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649612    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649615    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649618    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649621    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649623    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649627    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649629    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649632    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649634    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:01:31.650493 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649636    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649639    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649641    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649643    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649646    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649648    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649651    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649653    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649655    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649658    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649660    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649662    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649665    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649667    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649670    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649673    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649675    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649678    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649680    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649683    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:01:31.650966 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649686    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649688    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649692    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649696    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649699    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649702    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649705    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649708    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649712    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649714    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649717    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649719    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649722    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649725    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649729    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649732    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649734    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649737    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649740    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:01:31.651443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.649742    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650962    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650970    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650975    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650979    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650982    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650985    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650988    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650991    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650994    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650996    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.650999    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651001    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651003    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651007    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651009    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651012    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651015    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651018    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651021    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:01:31.651930 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651023    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651026    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651028    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651031    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651033    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651036    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651038    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651041    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651043    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651046    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651049    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651052    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651054    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651057    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651059    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651061    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651064    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651066    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651068    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651071    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:01:31.652498 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651073    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651076    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651078    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651080    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651083    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651085    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651088    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651091    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651094    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651097    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651099    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651102    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651105    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651107    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651110    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651112    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651115    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651119    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651122    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651126    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:01:31.653077 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651128    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651130    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651133    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651135    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651138    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651140    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651143    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651145    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651148    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651150    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651152    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651155    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651157    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651159    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651162    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651164    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651167    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651169    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651172    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:01:31.653656 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651174    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651178    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651182    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651185    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651188    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651191    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651194    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651196    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651199    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651202    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651204    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651206    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651209    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651211    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651214    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651216    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651219    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651221    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651223    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:01:31.654288 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651226    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651229    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651609    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651614    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651616    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651619    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651621    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651624    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651626    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651629    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651632    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651634    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651636    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651639    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651642    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651645    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651648    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651650    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651655    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:01:31.654829 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651658    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651661    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651664    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651667    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651669    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651675    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651678    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651681    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651684    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651686    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651689    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651691    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651694    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651697    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651700    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651703    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651707    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651710    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651713    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:01:31.655348 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651716    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651718    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651720    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651723    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651726    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651728    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651731    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651733    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651737    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651740    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651742    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651745    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651747    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651750    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651753    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651755    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651757    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651760    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651763    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651766    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:01:31.655905 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651768    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651771    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651773    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651775    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651778    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651787    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651789    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651792    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651795    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651797    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651799    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651802    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651804    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651807    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651809    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651812    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651814    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651817    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651819    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651822    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:01:31.656492 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651824    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651829    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651832    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651834    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651837    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651839    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651842    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651844    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651846    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651849    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651855    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651858    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651861    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651863    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651866    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651868    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651870    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651873    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651875    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651878    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:01:31.657033 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651880    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651882    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.651885    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651952    2579 flags.go:64] FLAG: --address="0.0.0.0"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651959    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651965    2579 flags.go:64] FLAG: --anonymous-auth="true"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651969    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651972    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651975    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651979    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651988    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651991    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651994    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.651998    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652001    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652005    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652008    2579 flags.go:64] FLAG: --cgroup-root=""
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652011    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652014    2579 flags.go:64] FLAG: --client-ca-file=""
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652016    2579 flags.go:64] FLAG: --cloud-provider="external"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652019    2579 flags.go:64] FLAG: --cluster-dns="[]"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652023    2579 flags.go:64] FLAG: --cluster-domain=""
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652027    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 27 17:01:31.657552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652029    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652032    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652035    2579 flags.go:64] FLAG: --container-log-max-files="5"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652039    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652042    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652045    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652048    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652051    2579 flags.go:64] FLAG: --contention-profiling="false"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652054    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652056    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652060    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652063    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652066    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652069    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652072    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652074    2579 flags.go:64] FLAG: --enable-load-reader="false"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652077    2579 flags.go:64] FLAG: --enable-server="true"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652080    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652083    2579 flags.go:64] FLAG: --event-burst="100"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652086    2579 flags.go:64] FLAG: --event-qps="50"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652089    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652092    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652097    2579 flags.go:64] FLAG: --eviction-hard=""
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652100    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 27 17:01:31.658146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652103    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652106    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652109    2579 flags.go:64] FLAG: --eviction-soft=""
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652112    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652115    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652118    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652121    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652124    2579 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652126    2579 flags.go:64] FLAG: --fail-swap-on="true"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652129    2579 flags.go:64] FLAG: --feature-gates=""
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652133    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652136    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652138    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652141    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652144    2579 flags.go:64] FLAG: --healthz-port="10248"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652147    2579 flags.go:64] FLAG: --help="false"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652150    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-194.ec2.internal"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652153    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652155    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652158    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652162    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652165    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652168    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 27 17:01:31.658793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652170    2579 flags.go:64] FLAG: --image-service-endpoint=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652173    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652175    2579 flags.go:64] FLAG: --kube-api-burst="100"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652178    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652181    2579 flags.go:64] FLAG: --kube-api-qps="50"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652184    2579 flags.go:64] FLAG: --kube-reserved=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652186    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652191    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652194    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652196    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652199    2579 flags.go:64] FLAG: --lock-file=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652202    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652205    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652207    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652212    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652215    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652218    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652220    2579 flags.go:64] FLAG: --logging-format="text"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652223    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652226    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652229    2579 flags.go:64] FLAG: --manifest-url=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652231    2579 flags.go:64] FLAG: --manifest-url-header=""
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652235    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652238    2579 flags.go:64] FLAG: --max-open-files="1000000"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652242    2579 flags.go:64] FLAG: --max-pods="110"
Jul 27 17:01:31.659387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652245    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652247    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652250    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652253    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652256    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652259    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652261    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652269    2579 flags.go:64] FLAG: --node-status-max-images="50"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652272    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652275    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652278    2579 flags.go:64] FLAG: --pod-cidr=""
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652281    2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652286    2579 flags.go:64] FLAG: --pod-manifest-path=""
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652289    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652292    2579 flags.go:64] FLAG: --pods-per-core="0"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652297    2579 flags.go:64] FLAG: --port="10250"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652300    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652302    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-098e01c0be2dc0144"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652305    2579 flags.go:64] FLAG: --qos-reserved=""
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652309    2579 flags.go:64] FLAG: --read-only-port="10255"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652312    2579 flags.go:64] FLAG: --register-node="true"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652314    2579 flags.go:64] FLAG: --register-with-taints=""
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652318    2579 flags.go:64] FLAG: --registry-burst="10"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652321    2579 flags.go:64] FLAG: --registry-qps="5"
Jul 27 17:01:31.660262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652323    2579 flags.go:64] FLAG: --reserved-cpus=""
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652326    2579 flags.go:64] FLAG: --reserved-memory=""
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652329    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652332    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652335    2579 flags.go:64] FLAG: --rotate-certificates="false"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652338    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652340    2579 flags.go:64] FLAG: --runonce="false"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652343    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652346    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652349    2579 flags.go:64] FLAG: --seccomp-default="false"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652351    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652354    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652357    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652360    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652363    2579 flags.go:64] FLAG: --storage-driver-password="root"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652366    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652369    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652371    2579 flags.go:64] FLAG: --storage-driver-user="root"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652374    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652377    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652380    2579 flags.go:64] FLAG: --system-cgroups=""
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652382    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652387    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652391    2579 flags.go:64] FLAG: --tls-cert-file=""
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652393    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 27 17:01:31.661172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652397    2579 flags.go:64] FLAG: --tls-min-version=""
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652400    2579 flags.go:64] FLAG: --tls-private-key-file=""
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652402    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652405    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652408    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652411    2579 flags.go:64] FLAG: --v="2"
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652416    2579 flags.go:64] FLAG: --version="false"
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652420    2579 flags.go:64] FLAG: --vmodule=""
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652423    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.652426    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652617    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652631    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652635    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652638    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652642    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652645    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652648    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652651    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652654    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652656    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652659    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652662    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:01:31.661815 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652665    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652668    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652671    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652673    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652676    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652679    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652689    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652692    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652695    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652697    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652700    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652703    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652705    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652708    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652711    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652713    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652716    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652718    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652721    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652724    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:01:31.662397 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652726    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652731    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652735    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652738    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652740    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652743    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652746    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652749    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652751    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652754    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652756    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652759    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652761    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652764    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652766    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652769    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652772    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652774    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652778    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:01:31.662911 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652788    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652791    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652795    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652799    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652802    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652805    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652808    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652811    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652813    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652816    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652818    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652821    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652824    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652827    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652832    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652834    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652837    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652839    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652842    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:01:31.663386 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652845    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652848    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652851    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652853    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652856    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652858    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652861    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652863    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652866    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652868    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652870    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652873    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652875    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652878    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652886    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652888    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652891    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652893    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652896    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652898    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:01:31.663879 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652901    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652903    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652905    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652908    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652910    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652914    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652917    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652920    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.652923    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.653438    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.661890    2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.661909    2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.661923    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.661929    2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 27 17:01:31.664371 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.661991    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.661996    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662000    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662003    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662007    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662009    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662012    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662015    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662018    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662021    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662023    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662026    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662028    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662031    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662033    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662035    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662038    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662041    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662043    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:01:31.664745 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662046    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662049    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662051    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662054    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662056    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662059    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662061    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662063    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662067    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662069    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662072    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662075    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662077    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662080    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662082    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662085    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662088    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662090    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662092    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:01:31.665215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662095    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662099    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662103    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662105    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662108    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662111    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662113    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662116    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662118    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662121    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662123    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662125    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662128    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662130    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662133    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662135    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662138    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662140    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662142    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:01:31.665757 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662145    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662147    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662150    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662153    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662158    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662161    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662164    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662167    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662169    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662172    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662175    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662177    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662180    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662183    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662185    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662189    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662192    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662195    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662198    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:01:31.666215 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662200    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662203    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662205    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662215    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662218    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662220    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662222    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662225    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662228    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662230    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662232    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662235    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662237    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662239    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662242    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662244    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662246    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662250    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662252    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662255    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:01:31.666721 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662258    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662260    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662263    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.662268    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662384    2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662388    2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662391    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662393    2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662396    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662399    2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662401    2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662404    2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662407    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662409    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662412    2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 27 17:01:31.667200 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662414    2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662425    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662428    2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662430    2579 feature_gate.go:328] unrecognized feature gate: Example
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662432    2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662435    2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662438    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662440    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662443    2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662445    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662448    2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662451    2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662454    2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662456    2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662459    2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662461    2579 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662464    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662466    2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662469    2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662485    2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 27 17:01:31.667595 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662488    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662490    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662493    2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662495    2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662497    2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662500    2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662502    2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662505    2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662508    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662510    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662513    2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662515    2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662518    2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662520    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662529    2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662532    2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662535    2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662537    2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662540    2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 27 17:01:31.668081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662542    2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662544    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662547    2579 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662550    2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662553    2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662555    2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662558    2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662562    2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662565    2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662568    2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662570    2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662573    2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662576    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662579    2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662582    2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662584    2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662586    2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662590    2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662593    2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 27 17:01:31.668568 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662595    2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662599    2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662601    2579 feature_gate.go:328] unrecognized feature gate: Example2
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662604    2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662606    2579 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662609    2579 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662612    2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662614    2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662617    2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662625    2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662628    2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662632    2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662635    2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662637    2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662640    2579 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662642    2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662644    2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662647    2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662650    2579 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662652    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 27 17:01:31.669041 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662654    2579 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662657    2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662659    2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662662    2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662664    2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662667    2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662670    2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662672    2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662674    2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:31.662677    2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.662682    2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.662787    2579 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.665055    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.666248    2579 server.go:1019] "Starting client certificate rotation"
Jul 27 17:01:31.669663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.666363    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:01:31.670008 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.666396    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 27 17:01:31.670008 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.668492    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:01:31.670887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.670875    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 27 17:01:31.681094 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.681073    2579 log.go:25] "Validated CRI v1 runtime API"
Jul 27 17:01:31.683078 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.683063    2579 log.go:25] "Validated CRI v1 image API"
Jul 27 17:01:31.686468 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.686454    2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 27 17:01:31.688567 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.688505    2579 fs.go:135] Filesystem UUIDs: map[31d445a1-08bd-4b20-8b7b-14fd5630490b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f9de901e-a38e-4d24-b8df-44eb184241b6:/dev/nvme0n1p4]
Jul 27 17:01:31.688567 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.688522    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 27 17:01:31.694104 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.693983    2579 manager.go:216] Machine: {Timestamp:2026-07-27 17:01:31.693463317 +0000 UTC m=+0.278824865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101730 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23692fcc2474a0835df07987abe910 SystemUUID:ec23692f-cc24-74a0-835d-f07987abe910 BootID:3527b6ae-0c4a-4beb-8aa1-fde3165c4525 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c4:50:31:a1:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c4:50:31:a1:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:ae:46:a0:75:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 27 17:01:31.694104 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.694091    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 27 17:01:31.694268 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.694167    2579 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 27 17:01:31.696156 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.696120    2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 27 17:01:31.696304 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.696163    2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-194.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 27 17:01:31.696834 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.696824    2579 topology_manager.go:138] "Creating topology manager with none policy"
Jul 27 17:01:31.696867 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.696837    2579 container_manager_linux.go:309] "Creating device plugin manager"
Jul 27 17:01:31.696867 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.696844    2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:01:31.697401 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.697390    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 27 17:01:31.698144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.698133    2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 27 17:01:31.700088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700068    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:01:31.700763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700751    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:01:31.700904 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700895    2579 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 27 17:01:31.700960 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700951    2579 kubelet.go:486] "Attempting to sync node with API server"
Jul 27 17:01:31.700996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700967    2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 27 17:01:31.700996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700980    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 27 17:01:31.700996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700989    2579 kubelet.go:398] "Adding apiserver pod source"
Jul 27 17:01:31.700996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.700997    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 27 17:01:31.702253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.702206    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:01:31.702319 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.702265    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 27 17:01:31.705118 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.705101    2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 27 17:01:31.706867 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.706854    2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 27 17:01:31.706914 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.706884    2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 27 17:01:31.708438 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708427    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 27 17:01:31.708494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708444    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 27 17:01:31.708494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708449    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 27 17:01:31.708494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708455    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 27 17:01:31.708494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708464    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 27 17:01:31.708494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708484    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 27 17:01:31.708494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708495    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 27 17:01:31.708654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708500    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 27 17:01:31.708654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708507    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 27 17:01:31.708654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708513    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 27 17:01:31.708654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708521    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 27 17:01:31.708654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.708527    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 27 17:01:31.709187 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.709177    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 27 17:01:31.709187 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.709186    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 27 17:01:31.712329 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.712311    2579 server.go:1268] "Started kubelet"
Jul 27 17:01:31.712401 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.712339    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-194.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 27 17:01:31.712619 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.712575    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 27 17:01:31.712703 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.712633    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 27 17:01:31.712959 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.712897    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 27 17:01:31.713044 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.713011    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 27 17:01:31.713096 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.713051    2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 27 17:01:31.713221 ip-10-0-132-194 systemd[1]: Started Kubernetes Kubelet.
Jul 27 17:01:31.715337 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.715320    2579 server.go:310] "Adding debug handlers to kubelet server"
Jul 27 17:01:31.715984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.715967    2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 27 17:01:31.717938 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.717922    2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 27 17:01:31.722566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.722549    2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 27 17:01:31.722566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.722559    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 27 17:01:31.722722 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.722703    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:31.722945 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.722107    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-194.ec2.internal.18c634320398163a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-194.ec2.internal,UID:ip-10-0-132-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-194.ec2.internal,},FirstTimestamp:2026-07-27 17:01:31.712288314 +0000 UTC m=+0.297649864,LastTimestamp:2026-07-27 17:01:31.712288314 +0000 UTC m=+0.297649864,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-194.ec2.internal,}"
Jul 27 17:01:31.723122 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723104    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 27 17:01:31.723200 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723108    2579 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 27 17:01:31.723200 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723147    2579 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 27 17:01:31.723304 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723293    2579 reconstruct.go:97] "Volume reconstruction finished"
Jul 27 17:01:31.723399 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723388    2579 reconciler.go:29] "Reconciler: start to sync state"
Jul 27 17:01:31.723806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723790    2579 factory.go:55] Registering systemd factory
Jul 27 17:01:31.723875 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.723809    2579 factory.go:223] Registration of the systemd container factory successfully
Jul 27 17:01:31.724051 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.724036    2579 factory.go:153] Registering CRI-O factory
Jul 27 17:01:31.724051 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.724052    2579 factory.go:223] Registration of the crio container factory successfully
Jul 27 17:01:31.724151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.724140    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 27 17:01:31.724184 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.724167    2579 factory.go:103] Registering Raw factory
Jul 27 17:01:31.724212 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.724187    2579 manager.go:1196] Started watching for new ooms in manager
Jul 27 17:01:31.724892 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.724881    2579 manager.go:349] Starting recovery of all containers
Jul 27 17:01:31.728042 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.728014    2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-194.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 27 17:01:31.728216 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.728190    2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 27 17:01:31.737264 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.737112    2579 manager.go:354] Recovery completed
Jul 27 17:01:31.738670 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.738652    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 27 17:01:31.741455 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.741444    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.743294 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743276    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.743361 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743308    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.743361 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743319    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:31.743871 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743850    2579 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 27 17:01:31.743871 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743867    2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 27 17:01:31.743984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743881    2579 state_mem.go:36] "Initialized new in-memory state store"
Jul 27 17:01:31.743984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.743876    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.744917 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.744898    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.744991 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.744942    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.744991 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.744955    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:31.745060 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.744987    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sr2cm"
Jul 27 17:01:31.746004 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.745924    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-194.ec2.internal.18c6343205713737  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-194.ec2.internal,UID:ip-10-0-132-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-194.ec2.internal,},FirstTimestamp:2026-07-27 17:01:31.743295287 +0000 UTC m=+0.328656834,LastTimestamp:2026-07-27 17:01:31.743295287 +0000 UTC m=+0.328656834,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-194.ec2.internal,}"
Jul 27 17:01:31.746562 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.746551    2579 policy_none.go:49] "None policy: Start"
Jul 27 17:01:31.746608 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.746571    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 27 17:01:31.746608 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.746583    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 27 17:01:31.748609 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.748598    2579 policy_none.go:47] "Start"
Jul 27 17:01:31.755754 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.755688    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-194.ec2.internal.18c6343205717eaa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-194.ec2.internal,UID:ip-10-0-132-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-194.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-194.ec2.internal,},FirstTimestamp:2026-07-27 17:01:31.743313578 +0000 UTC m=+0.328675125,LastTimestamp:2026-07-27 17:01:31.743313578 +0000 UTC m=+0.328675125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-194.ec2.internal,}"
Jul 27 17:01:31.758840 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.758783    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-194.ec2.internal.18c634320571a4fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-194.ec2.internal,UID:ip-10-0-132-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-194.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-194.ec2.internal,},FirstTimestamp:2026-07-27 17:01:31.743323387 +0000 UTC m=+0.328684934,LastTimestamp:2026-07-27 17:01:31.743323387 +0000 UTC m=+0.328684934,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-194.ec2.internal,}"
Jul 27 17:01:31.764541 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.764460    2579 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-194.ec2.internal.18c6343205713737\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-194.ec2.internal.18c6343205713737  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-194.ec2.internal,UID:ip-10-0-132-194.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-194.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-194.ec2.internal,},FirstTimestamp:2026-07-27 17:01:31.743295287 +0000 UTC m=+0.328656834,LastTimestamp:2026-07-27 17:01:31.744915915 +0000 UTC m=+0.330277469,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-194.ec2.internal,}"
Jul 27 17:01:31.765764 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.765744    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sr2cm"
Jul 27 17:01:31.789015 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789001    2579 manager.go:337] "Starting Device Plugin manager"
Jul 27 17:01:31.789101 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.789032    2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 27 17:01:31.789101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789042    2579 server.go:87] "Starting device plugin registration server"
Jul 27 17:01:31.789283 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789271    2579 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 27 17:01:31.789332 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789285    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 27 17:01:31.789407 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789391    2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 27 17:01:31.789910 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789517    2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 27 17:01:31.789910 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.789531    2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 27 17:01:31.790040 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.789979    2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 27 17:01:31.790040 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.790032    2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:31.885223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.885184    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 27 17:01:31.887516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.886532    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 27 17:01:31.887516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.886579    2579 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 27 17:01:31.887516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.886605    2579 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 27 17:01:31.887516 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.886649    2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 27 17:01:31.889365 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.889348    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.889505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.889391    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:01:31.890156 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.890136    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.890226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.890184    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.890226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.890203    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:31.890289 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.890232    2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-194.ec2.internal"
Jul 27 17:01:31.897158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.897110    2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-194.ec2.internal"
Jul 27 17:01:31.897158 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.897132    2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-194.ec2.internal\": node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:31.915070 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:31.915047    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:31.986755 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.986704    2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"]
Jul 27 17:01:31.986913 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.986792    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.988362 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.988338    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.988362 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.988364    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.988529 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.988381    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:31.990677 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.990665    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.990810 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.990792    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:31.991378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.991313    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.992959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.991898    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.992959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.991927    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.992959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.991948    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:31.992959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.992523    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.992959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.992550    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.992959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.992560    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:31.994371 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.994354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:31.994425 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.994383    2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 27 17:01:31.995126 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.995112    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientMemory"
Jul 27 17:01:31.995181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.995140    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasNoDiskPressure"
Jul 27 17:01:31.995181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:31.995149    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeHasSufficientPID"
Jul 27 17:01:32.009019 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.009000    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-194.ec2.internal\" not found" node="ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.012946 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.012931    2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-194.ec2.internal\" not found" node="ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.015144 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.015126    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:32.025447 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.025428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2bc28443e05da169c661f9a59da0dabc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal\" (UID: \"2bc28443e05da169c661f9a59da0dabc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.025549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.025459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bc28443e05da169c661f9a59da0dabc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal\" (UID: \"2bc28443e05da169c661f9a59da0dabc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.025549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.025501    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034461a4911a1521e502eca07e06f81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-194.ec2.internal\" (UID: \"1034461a4911a1521e502eca07e06f81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.115710 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.115678    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:32.126039 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.126012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2bc28443e05da169c661f9a59da0dabc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal\" (UID: \"2bc28443e05da169c661f9a59da0dabc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.126135 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.126042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bc28443e05da169c661f9a59da0dabc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal\" (UID: \"2bc28443e05da169c661f9a59da0dabc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.126135 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.126061    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034461a4911a1521e502eca07e06f81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-194.ec2.internal\" (UID: \"1034461a4911a1521e502eca07e06f81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.126135 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.126104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1034461a4911a1521e502eca07e06f81-config\") pod \"kube-apiserver-proxy-ip-10-0-132-194.ec2.internal\" (UID: \"1034461a4911a1521e502eca07e06f81\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.126252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.126131    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bc28443e05da169c661f9a59da0dabc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal\" (UID: \"2bc28443e05da169c661f9a59da0dabc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.126252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.126132    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2bc28443e05da169c661f9a59da0dabc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal\" (UID: \"2bc28443e05da169c661f9a59da0dabc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.216545 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.216454    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:32.312004 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.311957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.315788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.315759    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.316952 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.316841    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:32.417487 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.417449    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:32.518041 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.517951    2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-194.ec2.internal\" not found"
Jul 27 17:01:32.595116 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.595089    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:01:32.623841 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.623819    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.635719 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.635698    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:01:32.637272 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.637256    2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal"
Jul 27 17:01:32.647652 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.647625    2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 27 17:01:32.667253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.667225    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 27 17:01:32.667374 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.667356    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:01:32.667413 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.667377    2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 27 17:01:32.703318 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.703282    2579 apiserver.go:52] "Watching apiserver"
Jul 27 17:01:32.723299 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.723267    2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 27 17:01:32.723447 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.723268    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 27 17:01:32.723746 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.723726    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-88gv7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal","openshift-multus/multus-additional-cni-plugins-52wdr","openshift-multus/network-metrics-daemon-v6bp7","openshift-network-diagnostics/network-check-target-q46rj","openshift-network-operator/iptables-alerter-v9npj","openshift-ovn-kubernetes/ovnkube-node-bd48h","kube-system/konnectivity-agent-zxf8h","kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal","openshift-cluster-node-tuning-operator/tuned-4wmc2","openshift-dns/node-resolver-5brb2","openshift-image-registry/node-ca-5hgbt","openshift-multus/multus-5k9gw"]
Jul 27 17:01:32.726427 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.726412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.728780 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.728764    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.729813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.729798    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.729949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.729933    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sz2bz\""
Jul 27 17:01:32.731101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.731083    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.731186 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.731171    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.731787 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.731770    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 27 17:01:32.733121 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.733107    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.733442 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.733430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.733806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.733790    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.733886 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.733831    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 27 17:01:32.734013 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.733993    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 27 17:01:32.734380 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.734354    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.734449 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.734389    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 27 17:01:32.734449 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.734430    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tjst9\""
Jul 27 17:01:32.734574 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.734393    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nw7k9\""
Jul 27 17:01:32.734574 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.734454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.737731 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.737704    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:32.737849 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.737777    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:32.738315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.738299    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r2twh\""
Jul 27 17:01:32.738411 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.738338    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 27 17:01:32.738602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.738587    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 27 17:01:32.740533 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.740518    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:32.740631 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.740588    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:32.743166 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.742886    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.743353 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.743331    2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 27 17:01:32.745335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.745320    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.745403 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.745380    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:32.745981 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.745961    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 27 17:01:32.746061 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.745962    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 27 17:01:32.746106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.746090    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-krnrc\""
Jul 27 17:01:32.747816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.747802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.750296 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.750274    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.750665 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.750644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ltmlm\""
Jul 27 17:01:32.750786 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.750771    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.750960 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.750940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 27 17:01:32.751036 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.751022    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.752614 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.752600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.754724 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.754704    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.754999 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.754976    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.755316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.755291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 27 17:01:32.755316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.755291    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.755440 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.755348    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 27 17:01:32.755440 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.755293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 27 17:01:32.755534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.755514    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-d6n57\""
Jul 27 17:01:32.755633 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.755616    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.757143 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.757129    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hdt6r\""
Jul 27 17:01:32.757143 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.757135    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 27 17:01:32.757244 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.757138    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.758602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.758584    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 27 17:01:32.760020 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.760006    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pbnxf\""
Jul 27 17:01:32.760095 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.760080    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 27 17:01:32.771333 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.771290    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 16:56:32 +0000 UTC" deadline="2028-04-14 19:16:12.635057816 +0000 UTC"
Jul 27 17:01:32.771391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.771334    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15050h14m39.863726737s"
Jul 27 17:01:32.784319 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.784296    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-d8d6z"
Jul 27 17:01:32.791040 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.791021    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d8d6z"
Jul 27 17:01:32.824003 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.823981    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 27 17:01:32.830761 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830737    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/625d7bf4-f4eb-40d1-b524-e519680c5191-iptables-alerter-script\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.830848 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830772    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-hostroot\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.830848 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830801    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnl99\" (UniqueName: \"kubernetes.io/projected/158b0a3d-bf31-4102-9fd5-be98f076085c-kube-api-access-cnl99\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:32.830943 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-node-log\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.830943 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830886    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-cni-bin\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.830943 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.831057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.830994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-device-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.831057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-cnibin\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.831057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831025    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c-agent-certs\") pod \"konnectivity-agent-zxf8h\" (UID: \"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c\") " pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.831057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831039    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-cni-netd\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831065    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2h4z\" (UniqueName: \"kubernetes.io/projected/625d7bf4-f4eb-40d1-b524-e519680c5191-kube-api-access-w2h4z\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831086    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-system-cni-dir\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831109    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-cni-binary-copy\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831145    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-modprobe-d\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-system-cni-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831200    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-ovn\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-log-socket\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831249    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.831301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831276    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831343    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdnxt\" (UniqueName: \"kubernetes.io/projected/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-kube-api-access-vdnxt\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831369    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-systemd-units\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831403    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysctl-conf\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831432    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-sys\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286ed58f-538e-4c7e-8799-310708c75eed-tmp-dir\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831488    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-multus-certs\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831535    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-kubelet\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831564    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-env-overrides\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlwk5\" (UniqueName: \"kubernetes.io/projected/05546d5f-30c5-44f2-bf23-7bb187cc79b5-kube-api-access-xlwk5\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/467b5b33-2608-4734-8910-d45ba3de5566-dbus\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831646    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831681    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-run-netns\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-etc-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.831750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831750    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831769    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-socket-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-tmp\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831802    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lj8b\" (UniqueName: \"kubernetes.io/projected/122ef338-7a5d-409d-813d-b39955c3044a-kube-api-access-4lj8b\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysctl-d\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-var-lib-kubelet\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/286ed58f-538e-4c7e-8799-310708c75eed-hosts-file\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovnkube-config\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831917    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-k8s-cni-cncf-io\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831934    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-netns\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831949    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-etc-kubernetes\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831964    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/467b5b33-2608-4734-8910-d45ba3de5566-kubelet-config\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831980    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-kubelet\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.831994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/122ef338-7a5d-409d-813d-b39955c3044a-multus-daemon-config\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832010    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-slash\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-systemd\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832042    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05546d5f-30c5-44f2-bf23-7bb187cc79b5-host\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.832303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832056    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-systemd\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832072    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-cni-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832087    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-socket-dir-parent\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovn-node-metrics-cert\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832143    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovnkube-script-lib\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832161    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-sys-fs\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-run\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6g6\" (UniqueName: \"kubernetes.io/projected/286ed58f-538e-4c7e-8799-310708c75eed-kube-api-access-2h6g6\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-conf-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832267    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-registration-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-host\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832346    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-tuned\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832374    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832395    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c-konnectivity-ca\") pod \"konnectivity-agent-zxf8h\" (UID: \"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c\") " pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcqhb\" (UniqueName: \"kubernetes.io/projected/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-kube-api-access-zcqhb\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.832993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832438    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/05546d5f-30c5-44f2-bf23-7bb187cc79b5-serviceca\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-os-release\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832491    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvrs\" (UniqueName: \"kubernetes.io/projected/e2287062-988f-458d-a529-fb585b1a34eb-kube-api-access-pbvrs\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832507    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysconfig\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/122ef338-7a5d-409d-813d-b39955c3044a-cni-binary-copy\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-kubernetes\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-lib-modules\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-os-release\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832662    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-cni-bin\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832699    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-var-lib-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-cnibin\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-cni-multus\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832775    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/625d7bf4-f4eb-40d1-b524-e519680c5191-host-slash\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.833509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.832799    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpn9s\" (UniqueName: \"kubernetes.io/projected/652d605f-aa7e-4e83-9227-38245f2c2998-kube-api-access-kpn9s\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.881765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.881744    2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:01:32.884291 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.884275    2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:01:32.933357 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-system-cni-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.933357 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-ovn\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-log-socket\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-system-cni-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-log-socket\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933460    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-ovn\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933567    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdnxt\" (UniqueName: \"kubernetes.io/projected/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-kube-api-access-vdnxt\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.933581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-systemd-units\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.934011 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933598    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-tuning-conf-dir\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.934011 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933609    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysctl-conf\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.934011 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933632    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-sys\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.934011 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933656    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286ed58f-538e-4c7e-8799-310708c75eed-tmp-dir\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.934011 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933690    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-multus-certs\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.934011 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.933680    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-systemd-units\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.934316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934064    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-sys\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.934316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934106    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-multus-certs\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.934316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934299    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysctl-conf\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.934646 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934607    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.934646 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-kubelet\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.934817 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/286ed58f-538e-4c7e-8799-310708c75eed-tmp-dir\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.934817 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934799    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.934908 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-kubelet\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.934958 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934938    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-env-overrides\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.934976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlwk5\" (UniqueName: \"kubernetes.io/projected/05546d5f-30c5-44f2-bf23-7bb187cc79b5-kube-api-access-xlwk5\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/467b5b33-2608-4734-8910-d45ba3de5566-dbus\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935113    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-run-netns\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935142    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-etc-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935224    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-socket-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-tmp\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935303    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lj8b\" (UniqueName: \"kubernetes.io/projected/122ef338-7a5d-409d-813d-b39955c3044a-kube-api-access-4lj8b\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysctl-d\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-var-lib-kubelet\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935391    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/286ed58f-538e-4c7e-8799-310708c75eed-hosts-file\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935436    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovnkube-config\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935480    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-env-overrides\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935469    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-k8s-cni-cncf-io\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935644    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-socket-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.935704    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-k8s-cni-cncf-io\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.937062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936044    2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936156    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/467b5b33-2608-4734-8910-d45ba3de5566-dbus\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.936333    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.936431    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:01:33.436410513 +0000 UTC m=+2.021772065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936722    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-run-netns\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936807    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysctl-d\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-var-lib-kubelet\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936928    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/286ed58f-538e-4c7e-8799-310708c75eed-hosts-file\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-netns\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.936993    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-etc-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937010    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-etc-kubernetes\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937068    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-run-netns\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/467b5b33-2608-4734-8910-d45ba3de5566-kubelet-config\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937118    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-etc-kubernetes\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-kubelet\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937158    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/122ef338-7a5d-409d-813d-b39955c3044a-multus-daemon-config\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937184    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-slash\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937206    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-systemd\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937223    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05546d5f-30c5-44f2-bf23-7bb187cc79b5-host\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937231    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/467b5b33-2608-4734-8910-d45ba3de5566-kubelet-config\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-systemd\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937285    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-systemd\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937332    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-kubelet\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937322    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-cni-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-cni-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovnkube-config\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937500    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-run-systemd\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-slash\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937689    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05546d5f-30c5-44f2-bf23-7bb187cc79b5-host\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-socket-dir-parent\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovn-node-metrics-cert\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovnkube-script-lib\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937849    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-sys-fs\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937879    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-run\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6g6\" (UniqueName: \"kubernetes.io/projected/286ed58f-538e-4c7e-8799-310708c75eed-kube-api-access-2h6g6\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.938935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937929    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/122ef338-7a5d-409d-813d-b39955c3044a-multus-daemon-config\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937954    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-conf-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.937994    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-conf-dir\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-multus-socket-dir-parent\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-registration-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938137    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-registration-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-host\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938196    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-tuned\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938210    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-sys-fs\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938258    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-run-ovn-kubernetes\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c-konnectivity-ca\") pod \"konnectivity-agent-zxf8h\" (UID: \"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c\") " pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-host\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938599    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-run\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938675    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcqhb\" (UniqueName: \"kubernetes.io/projected/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-kube-api-access-zcqhb\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/05546d5f-30c5-44f2-bf23-7bb187cc79b5-serviceca\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.939872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.938998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-os-release\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939037    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c-konnectivity-ca\") pod \"konnectivity-agent-zxf8h\" (UID: \"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c\") " pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvrs\" (UniqueName: \"kubernetes.io/projected/e2287062-988f-458d-a529-fb585b1a34eb-kube-api-access-pbvrs\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939135    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-os-release\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939180    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysconfig\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/122ef338-7a5d-409d-813d-b39955c3044a-cni-binary-copy\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939311    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939329    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-sysconfig\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939351    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-kubernetes\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-lib-modules\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939437    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/05546d5f-30c5-44f2-bf23-7bb187cc79b5-serviceca\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939459    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-os-release\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939493    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-cni-bin\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939520    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-var-lib-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.939523    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-cnibin\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939603    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-kubernetes\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.940778 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-cni-multus\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.939657    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:33.439630392 +0000 UTC m=+2.024991947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939713    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-etc-selinux\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939721    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-cni-multus\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939729    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/625d7bf4-f4eb-40d1-b524-e519680c5191-host-slash\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939790    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpn9s\" (UniqueName: \"kubernetes.io/projected/652d605f-aa7e-4e83-9227-38245f2c2998-kube-api-access-kpn9s\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939814    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-os-release\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/625d7bf4-f4eb-40d1-b524-e519680c5191-iptables-alerter-script\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939870    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/625d7bf4-f4eb-40d1-b524-e519680c5191-host-slash\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939900    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-hostroot\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939946    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/122ef338-7a5d-409d-813d-b39955c3044a-cni-binary-copy\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.939962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnl99\" (UniqueName: \"kubernetes.io/projected/158b0a3d-bf31-4102-9fd5-be98f076085c-kube-api-access-cnl99\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-node-log\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940100    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-cni-bin\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940155    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-node-log\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.941697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-device-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940382    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-var-lib-openvswitch\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-cnibin\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940457    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-hostroot\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940461    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c-agent-certs\") pod \"konnectivity-agent-zxf8h\" (UID: \"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c\") " pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-device-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940617    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-cnibin\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940632    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-cnibin\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-cni-netd\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-cni-netd\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940749    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e2287062-988f-458d-a529-fb585b1a34eb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940872    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/625d7bf4-f4eb-40d1-b524-e519680c5191-iptables-alerter-script\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940867    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2h4z\" (UniqueName: \"kubernetes.io/projected/625d7bf4-f4eb-40d1-b524-e519680c5191-kube-api-access-w2h4z\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940884    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/122ef338-7a5d-409d-813d-b39955c3044a-host-var-lib-cni-bin\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940966    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-system-cni-dir\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.942474 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941030    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-cni-binary-copy\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941069    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-modprobe-d\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941105    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-tmp\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.940802    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-host-cni-bin\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941252    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovnkube-script-lib\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941327    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-modprobe-d\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941333    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-ovn-node-metrics-cert\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.941536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/652d605f-aa7e-4e83-9227-38245f2c2998-system-cni-dir\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.942384    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/652d605f-aa7e-4e83-9227-38245f2c2998-cni-binary-copy\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.942545    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-lib-modules\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.943236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.942703    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-etc-tuned\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.943543 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.943401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c-agent-certs\") pod \"konnectivity-agent-zxf8h\" (UID: \"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c\") " pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:32.955110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.955081    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlwk5\" (UniqueName: \"kubernetes.io/projected/05546d5f-30c5-44f2-bf23-7bb187cc79b5-kube-api-access-xlwk5\") pod \"node-ca-5hgbt\" (UID: \"05546d5f-30c5-44f2-bf23-7bb187cc79b5\") " pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:32.955723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.955705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdnxt\" (UniqueName: \"kubernetes.io/projected/1f35cd60-f79a-4a06-97c8-aef6d0851f6b-kube-api-access-vdnxt\") pod \"tuned-4wmc2\" (UID: \"1f35cd60-f79a-4a06-97c8-aef6d0851f6b\") " pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:32.958126 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.958100    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:32.958243 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.958132    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:32.958243 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.958147    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:32.958243 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:32.958205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:33.458188523 +0000 UTC m=+2.043550060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:32.960287 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.960260    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6g6\" (UniqueName: \"kubernetes.io/projected/286ed58f-538e-4c7e-8799-310708c75eed-kube-api-access-2h6g6\") pod \"node-resolver-5brb2\" (UID: \"286ed58f-538e-4c7e-8799-310708c75eed\") " pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:32.960414 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.960392    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2h4z\" (UniqueName: \"kubernetes.io/projected/625d7bf4-f4eb-40d1-b524-e519680c5191-kube-api-access-w2h4z\") pod \"iptables-alerter-v9npj\" (UID: \"625d7bf4-f4eb-40d1-b524-e519680c5191\") " pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:32.960597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.960580    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnl99\" (UniqueName: \"kubernetes.io/projected/158b0a3d-bf31-4102-9fd5-be98f076085c-kube-api-access-cnl99\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:32.961988 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.961965    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpn9s\" (UniqueName: \"kubernetes.io/projected/652d605f-aa7e-4e83-9227-38245f2c2998-kube-api-access-kpn9s\") pod \"multus-additional-cni-plugins-52wdr\" (UID: \"652d605f-aa7e-4e83-9227-38245f2c2998\") " pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:32.962968 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.962951    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lj8b\" (UniqueName: \"kubernetes.io/projected/122ef338-7a5d-409d-813d-b39955c3044a-kube-api-access-4lj8b\") pod \"multus-5k9gw\" (UID: \"122ef338-7a5d-409d-813d-b39955c3044a\") " pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:32.966814 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.966791    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvrs\" (UniqueName: \"kubernetes.io/projected/e2287062-988f-458d-a529-fb585b1a34eb-kube-api-access-pbvrs\") pod \"aws-ebs-csi-driver-node-spqqt\" (UID: \"e2287062-988f-458d-a529-fb585b1a34eb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:32.968594 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:32.968574    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcqhb\" (UniqueName: \"kubernetes.io/projected/cb742a4a-419b-4f76-a3b9-06f56c3b4fc6-kube-api-access-zcqhb\") pod \"ovnkube-node-bd48h\" (UID: \"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6\") " pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:33.048785 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.048701    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-v9npj"
Jul 27 17:01:33.069382 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.069354    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5hgbt"
Jul 27 17:01:33.086241 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.086218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-5k9gw"
Jul 27 17:01:33.103139 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.103113    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-52wdr"
Jul 27 17:01:33.125452 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.125422    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:33.141807 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.141785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt"
Jul 27 17:01:33.158419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.158394    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:33.176206 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.176182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4wmc2"
Jul 27 17:01:33.180138 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.180112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5brb2"
Jul 27 17:01:33.445843 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.445736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:33.445843 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.445817    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:33.446061 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.445889    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:33.446061 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.445919    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:33.446061 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.445948    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:01:34.445933497 +0000 UTC m=+3.031295033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:33.446061 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.445972    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:34.445955717 +0000 UTC m=+3.031317272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:33.546726 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.546697    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:33.546931 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.546870    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:33.546931 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.546891    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:33.546931 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.546901    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:33.547058 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:33.546956    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:34.546938265 +0000 UTC m=+3.132299807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:33.729083 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.729023    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1034461a4911a1521e502eca07e06f81.slice/crio-5bfbec5f652c804e2ff019c970db3ce3a26d440513e39d8d3c46f6ff69e24e35 WatchSource:0}: Error finding container 5bfbec5f652c804e2ff019c970db3ce3a26d440513e39d8d3c46f6ff69e24e35: Status 404 returned error can't find the container with id 5bfbec5f652c804e2ff019c970db3ce3a26d440513e39d8d3c46f6ff69e24e35
Jul 27 17:01:33.729698 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.729441    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286ed58f_538e_4c7e_8799_310708c75eed.slice/crio-e1f9fcc413d00bc3fe82c3d2162e0b9a51f750e5eeb0e6ffbeff7b1ad1f68928 WatchSource:0}: Error finding container e1f9fcc413d00bc3fe82c3d2162e0b9a51f750e5eeb0e6ffbeff7b1ad1f68928: Status 404 returned error can't find the container with id e1f9fcc413d00bc3fe82c3d2162e0b9a51f750e5eeb0e6ffbeff7b1ad1f68928
Jul 27 17:01:33.730158 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.730125    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122ef338_7a5d_409d_813d_b39955c3044a.slice/crio-5d47fbdce79f627ca009362838acd226fb02a153adcab808811f4dd9120172a8 WatchSource:0}: Error finding container 5d47fbdce79f627ca009362838acd226fb02a153adcab808811f4dd9120172a8: Status 404 returned error can't find the container with id 5d47fbdce79f627ca009362838acd226fb02a153adcab808811f4dd9120172a8
Jul 27 17:01:33.730858 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.730792    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625d7bf4_f4eb_40d1_b524_e519680c5191.slice/crio-647f96a6b6896cb192fbe842b76916cdba04b77fab6d47134d0bc82d1af37126 WatchSource:0}: Error finding container 647f96a6b6896cb192fbe842b76916cdba04b77fab6d47134d0bc82d1af37126: Status 404 returned error can't find the container with id 647f96a6b6896cb192fbe842b76916cdba04b77fab6d47134d0bc82d1af37126
Jul 27 17:01:33.733600 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.733542    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod652d605f_aa7e_4e83_9227_38245f2c2998.slice/crio-b162406883afc5cd751e2046df57dbbb34a5fa6a4263a4649ac734ebc206efd1 WatchSource:0}: Error finding container b162406883afc5cd751e2046df57dbbb34a5fa6a4263a4649ac734ebc206efd1: Status 404 returned error can't find the container with id b162406883afc5cd751e2046df57dbbb34a5fa6a4263a4649ac734ebc206efd1
Jul 27 17:01:33.736366 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.736352    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:01:33.758039 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.758008    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2287062_988f_458d_a529_fb585b1a34eb.slice/crio-72ab5692b98fdfe6ede16750315be787ada7e7f065619fed114eede51b443dff WatchSource:0}: Error finding container 72ab5692b98fdfe6ede16750315be787ada7e7f065619fed114eede51b443dff: Status 404 returned error can't find the container with id 72ab5692b98fdfe6ede16750315be787ada7e7f065619fed114eede51b443dff
Jul 27 17:01:33.794434 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.794361    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:56:33 +0000 UTC" deadline="2028-01-11 09:16:51.741990436 +0000 UTC"
Jul 27 17:01:33.794434 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.794424    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12784h15m17.94757152s"
Jul 27 17:01:33.818607 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.818569    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05546d5f_30c5_44f2_bf23_7bb187cc79b5.slice/crio-ec43975ddd796aac5f7437055787ab398282eede95c15f33aed8342bf32d246e WatchSource:0}: Error finding container ec43975ddd796aac5f7437055787ab398282eede95c15f33aed8342bf32d246e: Status 404 returned error can't find the container with id ec43975ddd796aac5f7437055787ab398282eede95c15f33aed8342bf32d246e
Jul 27 17:01:33.852722 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:33.852693    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb742a4a_419b_4f76_a3b9_06f56c3b4fc6.slice/crio-00c48551ab1ed23358ec739a88f77aeecd096fe78c141fc0300c7c044b95dbf1 WatchSource:0}: Error finding container 00c48551ab1ed23358ec739a88f77aeecd096fe78c141fc0300c7c044b95dbf1: Status 404 returned error can't find the container with id 00c48551ab1ed23358ec739a88f77aeecd096fe78c141fc0300c7c044b95dbf1
Jul 27 17:01:33.891648 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.891590    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerStarted","Data":"b162406883afc5cd751e2046df57dbbb34a5fa6a4263a4649ac734ebc206efd1"}
Jul 27 17:01:33.892708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.892679    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5brb2" event={"ID":"286ed58f-538e-4c7e-8799-310708c75eed","Type":"ContainerStarted","Data":"e1f9fcc413d00bc3fe82c3d2162e0b9a51f750e5eeb0e6ffbeff7b1ad1f68928"}
Jul 27 17:01:33.893773 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.893754    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5k9gw" event={"ID":"122ef338-7a5d-409d-813d-b39955c3044a","Type":"ContainerStarted","Data":"5d47fbdce79f627ca009362838acd226fb02a153adcab808811f4dd9120172a8"}
Jul 27 17:01:33.894876 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.894853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal" event={"ID":"1034461a4911a1521e502eca07e06f81","Type":"ContainerStarted","Data":"5bfbec5f652c804e2ff019c970db3ce3a26d440513e39d8d3c46f6ff69e24e35"}
Jul 27 17:01:33.895924 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.895905    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9npj" event={"ID":"625d7bf4-f4eb-40d1-b524-e519680c5191","Type":"ContainerStarted","Data":"647f96a6b6896cb192fbe842b76916cdba04b77fab6d47134d0bc82d1af37126"}
Jul 27 17:01:33.897154 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.897133    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"00c48551ab1ed23358ec739a88f77aeecd096fe78c141fc0300c7c044b95dbf1"}
Jul 27 17:01:33.898236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.898217    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5hgbt" event={"ID":"05546d5f-30c5-44f2-bf23-7bb187cc79b5","Type":"ContainerStarted","Data":"ec43975ddd796aac5f7437055787ab398282eede95c15f33aed8342bf32d246e"}
Jul 27 17:01:33.899315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:33.899288    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt" event={"ID":"e2287062-988f-458d-a529-fb585b1a34eb","Type":"ContainerStarted","Data":"72ab5692b98fdfe6ede16750315be787ada7e7f065619fed114eede51b443dff"}
Jul 27 17:01:34.006022 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.005936    2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:01:34.273079 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:34.273037    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc28443e05da169c661f9a59da0dabc.slice/crio-d9089fc774672198a92b84f78a6cf3e02bf170dbbfcc28005eef28c6674b93c5 WatchSource:0}: Error finding container d9089fc774672198a92b84f78a6cf3e02bf170dbbfcc28005eef28c6674b93c5: Status 404 returned error can't find the container with id d9089fc774672198a92b84f78a6cf3e02bf170dbbfcc28005eef28c6674b93c5
Jul 27 17:01:34.455506 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.454811    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:34.455506 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.454895    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:34.455506 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.455002    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:34.455506 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.455051    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:36.455037353 +0000 UTC m=+5.040398893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:34.455506 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.455411    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:34.455506 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.455471    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:01:36.455460901 +0000 UTC m=+5.040822436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:34.555568 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.555470    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:34.555741 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.555676    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:34.555741 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.555700    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:34.555741 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.555713    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:34.555908 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.555778    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:36.555759191 +0000 UTC m=+5.141120751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:34.618280 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:01:34.618216    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ea4d62_cc63_43b1_b2a2_3e3cdabbed9c.slice/crio-0e790256c0d6f99a4125962f398447588862e0af9d461f08d2cdd077fb98ead3 WatchSource:0}: Error finding container 0e790256c0d6f99a4125962f398447588862e0af9d461f08d2cdd077fb98ead3: Status 404 returned error can't find the container with id 0e790256c0d6f99a4125962f398447588862e0af9d461f08d2cdd077fb98ead3
Jul 27 17:01:34.794981 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.794928    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 16:56:33 +0000 UTC" deadline="2028-04-20 04:24:14.581485956 +0000 UTC"
Jul 27 17:01:34.794981 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.794976    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15179h22m39.786514751s"
Jul 27 17:01:34.887038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.886923    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:34.887224 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.887076    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:34.887619 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.887598    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:34.887743 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.887721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:34.887804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.887785    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:34.887859 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:34.887841    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:34.903957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.903907    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxf8h" event={"ID":"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c","Type":"ContainerStarted","Data":"0e790256c0d6f99a4125962f398447588862e0af9d461f08d2cdd077fb98ead3"}
Jul 27 17:01:34.905881 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:34.905853    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal" event={"ID":"2bc28443e05da169c661f9a59da0dabc","Type":"ContainerStarted","Data":"d9089fc774672198a92b84f78a6cf3e02bf170dbbfcc28005eef28c6674b93c5"}
Jul 27 17:01:35.850565 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:35.850533    2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 27 17:01:36.471793 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:36.471748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:36.472000 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:36.471821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:36.472000 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.471945    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:36.472000 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.471976    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:36.472132 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.472014    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:40.471998929 +0000 UTC m=+9.057360469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:36.472132 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.472041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:01:40.472022307 +0000 UTC m=+9.057383849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:36.572876 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:36.572826    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:36.573104 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.573045    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:36.573104 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.573074    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:36.573104 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.573089    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:36.573291 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.573163    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:40.573143213 +0000 UTC m=+9.158504769 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:36.887144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:36.887006    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:36.887144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:36.887031    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:36.887144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:36.887010    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:36.887700 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.887175    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:36.887700 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.887258    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:36.887700 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:36.887365    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:38.887274 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:38.887188    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:38.887777 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:38.887333    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:38.887777 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:38.887405    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:38.887777 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:38.887479    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:38.887777 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:38.887563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:38.887777 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:38.887655    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:40.507137 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:40.507095    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:40.507571 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:40.507199    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:40.507571 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.507339    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:40.507571 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.507430    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:48.507402667 +0000 UTC m=+17.092764211 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:40.507571 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.507510    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:40.507571 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.507565    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:01:48.507552097 +0000 UTC m=+17.092913639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:40.607838 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:40.607761    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:40.608022 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.608004    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:40.608071 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.608034    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:40.608071 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.608050    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:40.608139 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.608112    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:01:48.608091937 +0000 UTC m=+17.193453496 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:40.888661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:40.887786    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:40.888661 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.887912    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:40.888661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:40.888318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:40.888661 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.888426    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:40.888661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:40.888510    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:40.888661 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:40.888586    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:42.887181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.886930    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:42.887581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.886986    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:42.887581 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:42.887285    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:42.887581 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:42.887343    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:42.887581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.886989    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:42.887581 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:42.887457    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:42.926555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.926128    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5brb2" event={"ID":"286ed58f-538e-4c7e-8799-310708c75eed","Type":"ContainerStarted","Data":"f0422b65a9f14fea56ff2efee0d79f05e1f98c9ea56abd3d501a4409939f456f"}
Jul 27 17:01:42.927876 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.927845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt" event={"ID":"e2287062-988f-458d-a529-fb585b1a34eb","Type":"ContainerStarted","Data":"a89e938eb61711cf993b00f729efef08df1455693815589ae58ad24909fd7d91"}
Jul 27 17:01:42.929705 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.929682    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wmc2" event={"ID":"1f35cd60-f79a-4a06-97c8-aef6d0851f6b","Type":"ContainerStarted","Data":"8b28c456627b1d8065ef5376ecb68fa9f883c2479bf74e733a1400561d754976"}
Jul 27 17:01:42.943034 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:42.942966    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5brb2" podStartSLOduration=4.41700531 podStartE2EDuration="10.942948125s" totalImagesPullingTime="6.525942815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:42.942120292 +0000 UTC m=+11.527481849" watchObservedRunningTime="2026-07-27 17:01:42.942948125 +0000 UTC m=+11.528309683"
Jul 27 17:01:44.887343 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:44.887313    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:44.887763 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:44.887435    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:44.887837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:44.887818    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:44.887937 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:44.887912    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:44.887981 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:44.887951    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:44.888031 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:44.887995    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:44.935694 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:44.935631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-v9npj" event={"ID":"625d7bf4-f4eb-40d1-b524-e519680c5191","Type":"ContainerStarted","Data":"5cf337500141e9302a8fd1776ca5291e235a22c4578bffc96ff5114eba4eaaec"}
Jul 27 17:01:46.887315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:46.886839    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:46.887315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:46.886964    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:46.887315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:46.886969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:46.887315 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:46.886963    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:46.887315 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:46.887112    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:46.887315 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:46.887216    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:48.572189 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:48.572148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:48.572724 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:48.572227    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:48.572724 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.572337    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:48.572724 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.572362    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:48.572724 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.572403    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:04.572383589 +0000 UTC m=+33.157745204 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:01:48.572724 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.572429    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:04.572410606 +0000 UTC m=+33.157772165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:01:48.672775 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:48.672736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:48.672956 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.672903    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:01:48.672956 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.672921    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:01:48.672956 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.672933    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:48.673111 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.672989    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:04.672971441 +0000 UTC m=+33.258332992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:01:48.887680 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:48.887506    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:48.887843 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:48.887725    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:48.887843 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.887736    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:48.887843 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:48.887795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:48.888090 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.887900    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:48.888090 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:48.888006    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:50.887622 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:50.887581    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:50.888192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:50.887600    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:50.888192 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:50.887712    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:50.888192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:50.887729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:50.888192 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:50.887853    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:50.888192 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:50.887951    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:51.950745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:51.950564    2579 generic.go:356] "Generic (PLEG): container finished" podID="652d605f-aa7e-4e83-9227-38245f2c2998" containerID="547bab8adf3b685f3ce24b32db4f0828a62674b97eb94cfe623965816be1575a" exitCode=0
Jul 27 17:01:51.951283 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:51.950660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerDied","Data":"547bab8adf3b685f3ce24b32db4f0828a62674b97eb94cfe623965816be1575a"}
Jul 27 17:01:51.952625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:51.952595    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5hgbt" event={"ID":"05546d5f-30c5-44f2-bf23-7bb187cc79b5","Type":"ContainerStarted","Data":"9f755a63dc005c8d18c5d38e44fa8827cbd48ff714670b708455e26dadb6fe7f"}
Jul 27 17:01:51.954275 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:51.954252    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zxf8h" event={"ID":"04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c","Type":"ContainerStarted","Data":"c9035a94d55a577ad6fa843f711805e1bfab553a7a69704c5359601117b49f65"}
Jul 27 17:01:51.975830 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:51.975768    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-v9npj" podStartSLOduration=13.450394083 podStartE2EDuration="19.975747991s" totalImagesPullingTime="6.525353908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:44.950909503 +0000 UTC m=+13.536271059" watchObservedRunningTime="2026-07-27 17:01:51.975747991 +0000 UTC m=+20.561109548"
Jul 27 17:01:51.991111 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:51.991056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5hgbt" podStartSLOduration=12.260324788 podStartE2EDuration="19.991033448s" totalImagesPullingTime="7.73070866s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:51.990489993 +0000 UTC m=+20.575851540" watchObservedRunningTime="2026-07-27 17:01:51.991033448 +0000 UTC m=+20.576395006"
Jul 27 17:01:52.004691 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:52.004645    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zxf8h" podStartSLOduration=12.183469493 podStartE2EDuration="20.004625734s" totalImagesPullingTime="7.821156241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:52.004260975 +0000 UTC m=+20.589622535" watchObservedRunningTime="2026-07-27 17:01:52.004625734 +0000 UTC m=+20.589987290"
Jul 27 17:01:52.887263 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:52.886852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:52.887263 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:52.886852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:52.887263 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:52.886954    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:52.887263 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:52.886852    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:52.887263 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:52.887010    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:52.887263 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:52.887128    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:53.220273 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.220191    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:53.220874 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.220860    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:53.400239 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.400200    2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 27 17:01:53.802335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.802089    2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T17:01:53.400219045Z","UUID":"e508bfdb-e767-445e-a427-4ee344f29517","Handler":null,"Name":"","Endpoint":""}
Jul 27 17:01:53.804167 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.804147    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 27 17:01:53.804292 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.804183    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 27 17:01:53.959745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.959707    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-5k9gw" event={"ID":"122ef338-7a5d-409d-813d-b39955c3044a","Type":"ContainerStarted","Data":"3f3cd35a42c138de2426b94c9e74731b59782d53e3a5dd09d7bb03e3253fdc9b"}
Jul 27 17:01:53.961057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.961027    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4wmc2" event={"ID":"1f35cd60-f79a-4a06-97c8-aef6d0851f6b","Type":"ContainerStarted","Data":"1c05f885747857fa3c19b0adefdc615d744eb625c4940e0c0446adefcc12d00e"}
Jul 27 17:01:53.964030 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.964008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"84c682d0e46b93b8c469b4ab83442ee2d617a333d4cc88a5af74b7ba004e0f4e"}
Jul 27 17:01:53.964153 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.964086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"b4a54114965c03c3917451df1c7c6d8d6e766592a1b924526b85f030eb6d1f9f"}
Jul 27 17:01:53.964153 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.964138    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"b8bf4cafb4831ba2969d69401c26b29533dc09777ae2c88b7373432e87b36d94"}
Jul 27 17:01:53.964265 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.964188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"c9fe3eef99e27374e7b5e12f39ed122662c0dd79ae06ae362c15ad5d10ee366a"}
Jul 27 17:01:53.964265 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.964235    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"2be105327ea0547ccf9b63fe70a9f3b1f58cc2bf846635428860bd587ca6e84b"}
Jul 27 17:01:53.964358 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.964284    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"dc1cc545b38965b9e44c600bb45e73eaf60f62be6920452ebff379acb7951708"}
Jul 27 17:01:53.965392 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.965370    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal" event={"ID":"1034461a4911a1521e502eca07e06f81","Type":"ContainerStarted","Data":"653a0178f49ed8681bd61c714b66a0b5c8d9da37533a0f70fb22932a72f2bf29"}
Jul 27 17:01:53.966681 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.966659    2579 generic.go:356] "Generic (PLEG): container finished" podID="2bc28443e05da169c661f9a59da0dabc" containerID="09a5f25eba1054ba3cc518822d2177f6d5d8cafaf8d6e24fb3dadbe5beb77cee" exitCode=0
Jul 27 17:01:53.966776 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.966738    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal" event={"ID":"2bc28443e05da169c661f9a59da0dabc","Type":"ContainerDied","Data":"09a5f25eba1054ba3cc518822d2177f6d5d8cafaf8d6e24fb3dadbe5beb77cee"}
Jul 27 17:01:53.968235 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.968216    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt" event={"ID":"e2287062-988f-458d-a529-fb585b1a34eb","Type":"ContainerStarted","Data":"3f801c2ad4ccb06373403fc62db5f3f5ab304e01214362d3e3853eb6892c28c2"}
Jul 27 17:01:53.980658 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:53.980574    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-5k9gw" podStartSLOduration=2.440041813 podStartE2EDuration="21.980560125s" totalImagesPullingTime="19.540518312s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:53.980051085 +0000 UTC m=+22.565412640" watchObservedRunningTime="2026-07-27 17:01:53.980560125 +0000 UTC m=+22.565921708"
Jul 27 17:01:54.019972 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.019919    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4wmc2" podStartSLOduration=11.1911726 podStartE2EDuration="22.019899381s" totalImagesPullingTime="10.828726781s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:54.002114359 +0000 UTC m=+22.587475907" watchObservedRunningTime="2026-07-27 17:01:54.019899381 +0000 UTC m=+22.605260937"
Jul 27 17:01:54.037193 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.037152    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-194.ec2.internal" podStartSLOduration=21.037137766 podStartE2EDuration="21.037137766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:54.036793025 +0000 UTC m=+22.622154592" watchObservedRunningTime="2026-07-27 17:01:54.037137766 +0000 UTC m=+22.622499321"
Jul 27 17:01:54.887440 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.887407    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:54.887440 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.887424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:54.888276 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.887408    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:54.888276 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:54.887555    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:54.888276 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:54.887612    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:54.888276 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:54.887717    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:54.973147 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.973111    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal" event={"ID":"2bc28443e05da169c661f9a59da0dabc","Type":"ContainerStarted","Data":"fbc7bfe9390f8d6e2227500d4654b05e492a5ef69760cea843bbb4318cbf1aba"}
Jul 27 17:01:54.973317 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.973132    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:01:54.994107 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:54.994056    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-194.ec2.internal" podStartSLOduration=21.994040363 podStartE2EDuration="21.994040363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:54.993835289 +0000 UTC m=+23.579196847" watchObservedRunningTime="2026-07-27 17:01:54.994040363 +0000 UTC m=+23.579401918"
Jul 27 17:01:55.979007 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:55.978808    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"385c26aebe997ee4546f9702396fa177f20f992452e94ee97238b5a062b594c2"}
Jul 27 17:01:55.981019 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:55.980991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt" event={"ID":"e2287062-988f-458d-a529-fb585b1a34eb","Type":"ContainerStarted","Data":"b8867de80dcbc2c1e014e4b3dd570081b187bb7f2ef3597654849a00debddc09"}
Jul 27 17:01:56.887677 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:56.887639    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:56.887862 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:56.887685    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:56.887862 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:56.887693    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:56.887862 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:56.887764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:56.888013 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:56.887888    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:56.888013 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:56.887991    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:58.354304 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.354009    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:58.355126 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.354350    2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 27 17:01:58.355126 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.354571    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zxf8h"
Jul 27 17:01:58.369314 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.369272    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-spqqt" podStartSLOduration=7.582492536 podStartE2EDuration="26.369260105s" totalImagesPullingTime="18.786767569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:56.00040901 +0000 UTC m=+24.585770592" watchObservedRunningTime="2026-07-27 17:01:58.369260105 +0000 UTC m=+26.954621661"
Jul 27 17:01:58.887570 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.887540    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:01:58.887717 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.887574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:01:58.887717 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.887574    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:01:58.887717 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:58.887642    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:01:58.887814 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:58.887721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:01:58.887845 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:01:58.887807    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:01:58.991055 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.991016    2579 generic.go:356] "Generic (PLEG): container finished" podID="652d605f-aa7e-4e83-9227-38245f2c2998" containerID="b5aa75e127744da22891b2661329498ab8aa07a8a8860f3beb89bd13d958308c" exitCode=0
Jul 27 17:01:58.991223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.991104    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerDied","Data":"b5aa75e127744da22891b2661329498ab8aa07a8a8860f3beb89bd13d958308c"}
Jul 27 17:01:58.995716 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.995681    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" event={"ID":"cb742a4a-419b-4f76-a3b9-06f56c3b4fc6","Type":"ContainerStarted","Data":"8081ded545defd2f760abe655ecdd5101b7aeb684b1844edbfaacf61fb7926ff"}
Jul 27 17:01:58.995957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.995940    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:58.996056 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.996012    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:58.996122 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:58.996062    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:59.012114 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:59.012093    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:59.012228 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:59.012195    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:01:59.040175 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:01:59.040135    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h" podStartSLOduration=7.619623498 podStartE2EDuration="27.040122876s" totalImagesPullingTime="19.420499378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:01:59.038744527 +0000 UTC m=+27.624106085" watchObservedRunningTime="2026-07-27 17:01:59.040122876 +0000 UTC m=+27.625484432"
Jul 27 17:02:00.266672 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:00.266639    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88gv7"]
Jul 27 17:02:00.267072 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:00.266768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:00.267072 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:00.266855    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:02:00.269296 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:00.269270    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6bp7"]
Jul 27 17:02:00.269580 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:00.269563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:02:00.269801 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:00.269764    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:02:00.270151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:00.270132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q46rj"]
Jul 27 17:02:00.270240 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:00.270227    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:00.270327 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:00.270306    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:02:01.888143 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:01.887955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:01.888501 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:01.888021    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:02:01.888501 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:01.888207    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:02:01.888501 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:01.888305    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:02:01.888501 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:01.888039    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:01.888501 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:01.888374    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:02:02.004445 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:02.004413    2579 generic.go:356] "Generic (PLEG): container finished" podID="652d605f-aa7e-4e83-9227-38245f2c2998" containerID="5abaa643d5ab5c43e7c3f7a6e7b5445cbc164cfff00c23a2a39623192d094b47" exitCode=0
Jul 27 17:02:02.004604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:02.004455    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerDied","Data":"5abaa643d5ab5c43e7c3f7a6e7b5445cbc164cfff00c23a2a39623192d094b47"}
Jul 27 17:02:03.009237 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:03.009139    2579 generic.go:356] "Generic (PLEG): container finished" podID="652d605f-aa7e-4e83-9227-38245f2c2998" containerID="1d14059365101f38f53ab52a3696b7372529ab54608e93f1909edfb0ab0ee2ab" exitCode=0
Jul 27 17:02:03.009237 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:03.009188    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerDied","Data":"1d14059365101f38f53ab52a3696b7372529ab54608e93f1909edfb0ab0ee2ab"}
Jul 27 17:02:03.887650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:03.887621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:03.887820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:03.887629    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:02:03.887820 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:03.887712    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-88gv7" podUID="467b5b33-2608-4734-8910-d45ba3de5566"
Jul 27 17:02:03.887820 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:03.887808    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-v6bp7" podUID="158b0a3d-bf31-4102-9fd5-be98f076085c"
Jul 27 17:02:03.887996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:03.887862    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:03.887996 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:03.887924    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-q46rj" podUID="ff986bd4-212e-4ef6-aa1c-6ee37386a7b3"
Jul 27 17:02:04.594654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:04.594619    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:02:04.595054 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:04.594701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:04.595054 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.594759    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:02:04.595054 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.594802    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 27 17:02:04.595054 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.594828    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:02:36.594808638 +0000 UTC m=+65.180170197 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 27 17:02:04.595054 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.594849    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret podName:467b5b33-2608-4734-8910-d45ba3de5566 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:36.594835393 +0000 UTC m=+65.180196934 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret") pod "global-pull-secret-syncer-88gv7" (UID: "467b5b33-2608-4734-8910-d45ba3de5566") : object "kube-system"/"original-pull-secret" not registered
Jul 27 17:02:04.695900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:04.695862    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:04.696078 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.696049    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 27 17:02:04.696078 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.696069    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 27 17:02:04.696078 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.696078    2579 projected.go:196] Error preparing data for projected volume kube-api-access-882lx for pod openshift-network-diagnostics/network-check-target-q46rj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:02:04.696222 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:04.696139    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx podName:ff986bd4-212e-4ef6-aa1c-6ee37386a7b3 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:36.696121464 +0000 UTC m=+65.281483002 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-882lx" (UniqueName: "kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx") pod "network-check-target-q46rj" (UID: "ff986bd4-212e-4ef6-aa1c-6ee37386a7b3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 27 17:02:05.785015 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.784983    2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-194.ec2.internal" event="NodeReady"
Jul 27 17:02:05.785550 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.785137    2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 27 17:02:05.835653 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.835592    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tzvrv"]
Jul 27 17:02:05.849814 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.849781    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-c5glj"]
Jul 27 17:02:05.849945 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.849892    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:05.852258 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.852212    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 27 17:02:05.852436 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.852402    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 27 17:02:05.852536 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.852444    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5sbpw\""
Jul 27 17:02:05.859751 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.859729    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tzvrv"]
Jul 27 17:02:05.859844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.859769    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c5glj"]
Jul 27 17:02:05.859901 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.859861    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:05.862801 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.862760    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 27 17:02:05.862893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.862872    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 27 17:02:05.863038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.863015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dmcwl\""
Jul 27 17:02:05.863136 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.863070    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 27 17:02:05.891215 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.891186    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:05.891354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.891196    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:02:05.891408 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.891199    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:05.893653 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.893630    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:02:05.893653 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.893643    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:02:05.893933 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.893916    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxtqx\""
Jul 27 17:02:05.894025 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.894008    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:02:05.894099 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.893922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c7drs\""
Jul 27 17:02:05.894148 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:05.894096    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:02:06.006633 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.006598    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-kube-api-access-c8jn6\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.006790 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.006702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-tmp-dir\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.006790 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.006734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tkvv\" (UniqueName: \"kubernetes.io/projected/da55db76-5640-4deb-916f-58b5c50c9f95-kube-api-access-8tkvv\") pod \"ingress-canary-c5glj\" (UID: \"da55db76-5640-4deb-916f-58b5c50c9f95\") " pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.006790 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.006787    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-config-volume\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.006944 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.006805    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.006944 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.006833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da55db76-5640-4deb-916f-58b5c50c9f95-cert\") pod \"ingress-canary-c5glj\" (UID: \"da55db76-5640-4deb-916f-58b5c50c9f95\") " pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.108131 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108092    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tkvv\" (UniqueName: \"kubernetes.io/projected/da55db76-5640-4deb-916f-58b5c50c9f95-kube-api-access-8tkvv\") pod \"ingress-canary-c5glj\" (UID: \"da55db76-5640-4deb-916f-58b5c50c9f95\") " pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.108291 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-config-volume\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.108405 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108380    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.108507 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108430    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da55db76-5640-4deb-916f-58b5c50c9f95-cert\") pod \"ingress-canary-c5glj\" (UID: \"da55db76-5640-4deb-916f-58b5c50c9f95\") " pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.108557 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-kube-api-access-c8jn6\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.108557 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:06.108546    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:06.108643 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-tmp-dir\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.108643 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:06.108610    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls podName:d51131f9-ae3d-48ce-94b7-ce04477ef8f1 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:06.608590778 +0000 UTC m=+35.193952318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls") pod "dns-default-tzvrv" (UID: "d51131f9-ae3d-48ce-94b7-ce04477ef8f1") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:06.108895 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108874    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-tmp-dir\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.108995 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.108977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-config-volume\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.113114 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.113093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da55db76-5640-4deb-916f-58b5c50c9f95-cert\") pod \"ingress-canary-c5glj\" (UID: \"da55db76-5640-4deb-916f-58b5c50c9f95\") " pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.119510 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.119464    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tkvv\" (UniqueName: \"kubernetes.io/projected/da55db76-5640-4deb-916f-58b5c50c9f95-kube-api-access-8tkvv\") pod \"ingress-canary-c5glj\" (UID: \"da55db76-5640-4deb-916f-58b5c50c9f95\") " pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.119814 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.119790    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8jn6\" (UniqueName: \"kubernetes.io/projected/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-kube-api-access-c8jn6\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.174589 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.174561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-c5glj"
Jul 27 17:02:06.611814 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:06.611779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:06.611993 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:06.611963    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:06.612052 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:06.612041    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls podName:d51131f9-ae3d-48ce-94b7-ce04477ef8f1 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:07.612018559 +0000 UTC m=+36.197380095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls") pod "dns-default-tzvrv" (UID: "d51131f9-ae3d-48ce-94b7-ce04477ef8f1") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:07.619221 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:07.619179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:07.619887 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:07.619345    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:07.619887 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:07.619427    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls podName:d51131f9-ae3d-48ce-94b7-ce04477ef8f1 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:09.61940437 +0000 UTC m=+38.204765908 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls") pod "dns-default-tzvrv" (UID: "d51131f9-ae3d-48ce-94b7-ce04477ef8f1") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:07.799675 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:07.799536    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-c5glj"]
Jul 27 17:02:07.802797 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:07.802774    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda55db76_5640_4deb_916f_58b5c50c9f95.slice/crio-7d94e870a0fc98c81335d3c3428564dc7eb3975236c5ec702df20e3c61436bee WatchSource:0}: Error finding container 7d94e870a0fc98c81335d3c3428564dc7eb3975236c5ec702df20e3c61436bee: Status 404 returned error can't find the container with id 7d94e870a0fc98c81335d3c3428564dc7eb3975236c5ec702df20e3c61436bee
Jul 27 17:02:08.021995 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:08.021960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c5glj" event={"ID":"da55db76-5640-4deb-916f-58b5c50c9f95","Type":"ContainerStarted","Data":"7d94e870a0fc98c81335d3c3428564dc7eb3975236c5ec702df20e3c61436bee"}
Jul 27 17:02:08.024587 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:08.024562    2579 generic.go:356] "Generic (PLEG): container finished" podID="652d605f-aa7e-4e83-9227-38245f2c2998" containerID="28ef09b52b0fb0e24fda0602a521aff77ee60c23cc6c2e649f67a52cd964a1df" exitCode=0
Jul 27 17:02:08.024713 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:08.024603    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerDied","Data":"28ef09b52b0fb0e24fda0602a521aff77ee60c23cc6c2e649f67a52cd964a1df"}
Jul 27 17:02:09.035591 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:09.035557    2579 generic.go:356] "Generic (PLEG): container finished" podID="652d605f-aa7e-4e83-9227-38245f2c2998" containerID="6f9c6c1ce0b61784cadbec099c45b2975bcd4b4f3aa388716328b6e7820f01ee" exitCode=0
Jul 27 17:02:09.036122 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:09.035607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerDied","Data":"6f9c6c1ce0b61784cadbec099c45b2975bcd4b4f3aa388716328b6e7820f01ee"}
Jul 27 17:02:09.639115 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:09.638909    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:09.639275 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:09.639064    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:09.639275 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:09.639238    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls podName:d51131f9-ae3d-48ce-94b7-ce04477ef8f1 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:13.639222839 +0000 UTC m=+42.224584373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls") pod "dns-default-tzvrv" (UID: "d51131f9-ae3d-48ce-94b7-ce04477ef8f1") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:10.039962 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:10.039923    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-c5glj" event={"ID":"da55db76-5640-4deb-916f-58b5c50c9f95","Type":"ContainerStarted","Data":"a8423b6e2c16573bea3ff08fcbde009285bf81e2602339b7fc63d2e6e8880b32"}
Jul 27 17:02:10.043287 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:10.043258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-52wdr" event={"ID":"652d605f-aa7e-4e83-9227-38245f2c2998","Type":"ContainerStarted","Data":"fb3a43fdc23c58065b68a40368f44083f296925603429c77775b06e374d7b88d"}
Jul 27 17:02:10.060581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:10.060524    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-c5glj" podStartSLOduration=2.925792431 podStartE2EDuration="5.060502437s" totalImagesPullingTime="2.134710006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:10.059760669 +0000 UTC m=+38.645122227" watchObservedRunningTime="2026-07-27 17:02:10.060502437 +0000 UTC m=+38.645863993"
Jul 27 17:02:10.091779 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:10.091730    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-52wdr" podStartSLOduration=15.928954842 podStartE2EDuration="38.091712435s" totalImagesPullingTime="22.162757593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:10.091467204 +0000 UTC m=+38.676828760" watchObservedRunningTime="2026-07-27 17:02:10.091712435 +0000 UTC m=+38.677073985"
Jul 27 17:02:13.668821 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:13.668778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:13.669274 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:13.668917    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:13.669274 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:13.668977    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls podName:d51131f9-ae3d-48ce-94b7-ce04477ef8f1 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:21.668961791 +0000 UTC m=+50.254323324 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls") pod "dns-default-tzvrv" (UID: "d51131f9-ae3d-48ce-94b7-ce04477ef8f1") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:18.035461 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.035425    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"]
Jul 27 17:02:18.094985 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.094950    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-d7xqf"]
Jul 27 17:02:18.095136 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.095080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"
Jul 27 17:02:18.097870 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.097844    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:02:18.098149 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.098125    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hwjff\""
Jul 27 17:02:18.098953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.098939    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 27 17:02:18.105340 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.105318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"]
Jul 27 17:02:18.105340 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.105351    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-d7xqf"]
Jul 27 17:02:18.105483 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.105432    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.108322 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.108302    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 27 17:02:18.108421 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.108350    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 27 17:02:18.108421 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.108370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 27 17:02:18.108554 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.108416    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 27 17:02:18.108554 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.108356    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5nwl4\""
Jul 27 17:02:18.113313 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.113295    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 27 17:02:18.144952 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.144920    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-57898f8749-pb7sc"]
Jul 27 17:02:18.170800 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.170763    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57898f8749-pb7sc"]
Jul 27 17:02:18.170951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.170882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.174624 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.174603    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 27 17:02:18.175060 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.175038    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 27 17:02:18.175166 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.175095    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 27 17:02:18.175354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.175338    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7zvk7\""
Jul 27 17:02:18.175426 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.175401    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 27 17:02:18.175426 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.175411    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 27 17:02:18.175559 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.175428    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 27 17:02:18.198527 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.198492    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrr6w\" (UniqueName: \"kubernetes.io/projected/c5b6f892-c394-4968-8128-c61aa6f328a2-kube-api-access-rrr6w\") pod \"volume-data-source-validator-549c6d54cf-q8p9z\" (UID: \"c5b6f892-c394-4968-8128-c61aa6f328a2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"
Jul 27 17:02:18.298998 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.298909    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-serving-cert\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.298998 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.298947    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-trusted-ca\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.298998 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.298965    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-default-certificate\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-config\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx7sm\" (UniqueName: \"kubernetes.io/projected/dc3e5f58-84bb-4406-b871-1dbdea1e2104-kube-api-access-vx7sm\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299118    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-stats-auth\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299137    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299171    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f55v2\" (UniqueName: \"kubernetes.io/projected/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-kube-api-access-f55v2\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrr6w\" (UniqueName: \"kubernetes.io/projected/c5b6f892-c394-4968-8128-c61aa6f328a2-kube-api-access-rrr6w\") pod \"volume-data-source-validator-549c6d54cf-q8p9z\" (UID: \"c5b6f892-c394-4968-8128-c61aa6f328a2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"
Jul 27 17:02:18.299231 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.299220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.313596 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.313561    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrr6w\" (UniqueName: \"kubernetes.io/projected/c5b6f892-c394-4968-8128-c61aa6f328a2-kube-api-access-rrr6w\") pod \"volume-data-source-validator-549c6d54cf-q8p9z\" (UID: \"c5b6f892-c394-4968-8128-c61aa6f328a2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"
Jul 27 17:02:18.400493 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.400493 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400500    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f55v2\" (UniqueName: \"kubernetes.io/projected/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-kube-api-access-f55v2\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-serving-cert\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400572    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-trusted-ca\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-default-certificate\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:18.400615    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:18.900590846 +0000 UTC m=+47.485952381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : configmap references non-existent config key: service-ca.crt
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:18.400669    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400676    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-config\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.400757 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:18.400744    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:18.900725792 +0000 UTC m=+47.486087356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : secret "router-metrics-certs-default" not found
Jul 27 17:02:18.401151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400819    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx7sm\" (UniqueName: \"kubernetes.io/projected/dc3e5f58-84bb-4406-b871-1dbdea1e2104-kube-api-access-vx7sm\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.401151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.400851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-stats-auth\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.401403 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.401366    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-config\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.401904 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.401879    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-trusted-ca\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.403262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.403238    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-serving-cert\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.403365 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.403352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-stats-auth\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.403404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.403351    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-default-certificate\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.404097 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.404077    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"
Jul 27 17:02:18.413654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.413631    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx7sm\" (UniqueName: \"kubernetes.io/projected/dc3e5f58-84bb-4406-b871-1dbdea1e2104-kube-api-access-vx7sm\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.414124 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.414102    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f55v2\" (UniqueName: \"kubernetes.io/projected/5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb-kube-api-access-f55v2\") pod \"console-operator-584dcb6b54-d7xqf\" (UID: \"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb\") " pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.414257 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.414243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:18.584252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.584146    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z"]
Jul 27 17:02:18.587514 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:18.587485    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b6f892_c394_4968_8128_c61aa6f328a2.slice/crio-a963a46706b9a2f161b61680230436c3e6d703345ed215152fddf160fe698d99 WatchSource:0}: Error finding container a963a46706b9a2f161b61680230436c3e6d703345ed215152fddf160fe698d99: Status 404 returned error can't find the container with id a963a46706b9a2f161b61680230436c3e6d703345ed215152fddf160fe698d99
Jul 27 17:02:18.604178 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.604151    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-d7xqf"]
Jul 27 17:02:18.609356 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:18.609330    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ef78ccd_3ab2_407f_92d3_6ec2ff5e2bcb.slice/crio-17aac427f35d271c791d9c849b9b2d00acf5ce442daea71a4a559fe2093acd7d WatchSource:0}: Error finding container 17aac427f35d271c791d9c849b9b2d00acf5ce442daea71a4a559fe2093acd7d: Status 404 returned error can't find the container with id 17aac427f35d271c791d9c849b9b2d00acf5ce442daea71a4a559fe2093acd7d
Jul 27 17:02:18.904243 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.904151    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.904243 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:18.904208    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:18.904500 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:18.904342    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 17:02:18.904500 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:18.904370    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:19.904345912 +0000 UTC m=+48.489707449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : configmap references non-existent config key: service-ca.crt
Jul 27 17:02:18.904500 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:18.904398    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:19.904388855 +0000 UTC m=+48.489750392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : secret "router-metrics-certs-default" not found
Jul 27 17:02:19.066399 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:19.066357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z" event={"ID":"c5b6f892-c394-4968-8128-c61aa6f328a2","Type":"ContainerStarted","Data":"a963a46706b9a2f161b61680230436c3e6d703345ed215152fddf160fe698d99"}
Jul 27 17:02:19.067455 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:19.067432    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" event={"ID":"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb","Type":"ContainerStarted","Data":"17aac427f35d271c791d9c849b9b2d00acf5ce442daea71a4a559fe2093acd7d"}
Jul 27 17:02:19.915257 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:19.915216    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:19.915431 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:19.915298    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:19.915431 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:19.915357    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 17:02:19.915431 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:19.915419    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:21.915405275 +0000 UTC m=+50.500766810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : configmap references non-existent config key: service-ca.crt
Jul 27 17:02:19.915576 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:19.915434    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:21.91542759 +0000 UTC m=+50.500789124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : secret "router-metrics-certs-default" not found
Jul 27 17:02:21.726428 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:21.726394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:21.726808 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:21.726563    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 27 17:02:21.726808 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:21.726627    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls podName:d51131f9-ae3d-48ce-94b7-ce04477ef8f1 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:37.726612115 +0000 UTC m=+66.311973648 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls") pod "dns-default-tzvrv" (UID: "d51131f9-ae3d-48ce-94b7-ce04477ef8f1") : secret "dns-default-metrics-tls" not found
Jul 27 17:02:21.927672 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:21.927629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:21.927843 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:21.927686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:21.927843 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:21.927777    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 17:02:21.927843 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:21.927796    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.927777835 +0000 UTC m=+54.513139370 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : configmap references non-existent config key: service-ca.crt
Jul 27 17:02:21.927843 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:21.927821    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.927810439 +0000 UTC m=+54.513171973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : secret "router-metrics-certs-default" not found
Jul 27 17:02:22.179783 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.179754    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mz5cv"]
Jul 27 17:02:22.193663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.193643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.197135 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.197113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 27 17:02:22.197601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.197579    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hn8cb\""
Jul 27 17:02:22.198169 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.198153    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 27 17:02:22.198370 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.198155    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 27 17:02:22.198424 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.198373    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 27 17:02:22.202550 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.202523    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mz5cv"]
Jul 27 17:02:22.230160 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.230134    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.230260 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.230170    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.230260 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.230188    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzhfc\" (UniqueName: \"kubernetes.io/projected/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-api-access-lzhfc\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.230260 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.230211    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8fa5de4-86ff-43ff-923e-5a3bf987c540-data-volume\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.230260 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.230239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a8fa5de4-86ff-43ff-923e-5a3bf987c540-crio-socket\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.330628 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.330598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.330749 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.330639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.330749 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.330655    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzhfc\" (UniqueName: \"kubernetes.io/projected/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-api-access-lzhfc\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.330749 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.330674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8fa5de4-86ff-43ff-923e-5a3bf987c540-data-volume\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.330749 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.330695    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a8fa5de4-86ff-43ff-923e-5a3bf987c540-crio-socket\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.330749 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.330698    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.330903 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.330767    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:22.830752232 +0000 UTC m=+51.416113767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.330903 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.330783    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a8fa5de4-86ff-43ff-923e-5a3bf987c540-crio-socket\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.331010 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.330985    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.331010 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.331001    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a8fa5de4-86ff-43ff-923e-5a3bf987c540-data-volume\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.331083 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.331029    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:22.831017613 +0000 UTC m=+51.416379147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.341029 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.341000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzhfc\" (UniqueName: \"kubernetes.io/projected/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-api-access-lzhfc\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.833853 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.833807    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.834290 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:22.833921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:22.834290 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.833986    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.834290 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.834005    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:22.834290 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.834037    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:23.834022329 +0000 UTC m=+52.419383876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:22.834290 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:22.834074    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:23.834055501 +0000 UTC m=+52.419417055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:23.079088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.079059    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/0.log"
Jul 27 17:02:23.079263 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.079101    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb" containerID="34a5ee8fc97389734e33cfafd627ac6c1c996bb079baadc647e557412f39e630" exitCode=255
Jul 27 17:02:23.079263 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.079165    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" event={"ID":"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb","Type":"ContainerDied","Data":"34a5ee8fc97389734e33cfafd627ac6c1c996bb079baadc647e557412f39e630"}
Jul 27 17:02:23.079436 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.079422    2579 scope.go:117] "RemoveContainer" containerID="34a5ee8fc97389734e33cfafd627ac6c1c996bb079baadc647e557412f39e630"
Jul 27 17:02:23.080769 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.080745    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z" event={"ID":"c5b6f892-c394-4968-8128-c61aa6f328a2","Type":"ContainerStarted","Data":"2eff4545c1ae2390656352427df384895c87d9f11daa2ae838b8e80237cc6eb3"}
Jul 27 17:02:23.111284 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.111232    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-q8p9z" podStartSLOduration=1.511594315 podStartE2EDuration="5.111214933s" totalImagesPullingTime="3.599620618s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:23.111175307 +0000 UTC m=+51.696536864" watchObservedRunningTime="2026-07-27 17:02:23.111214933 +0000 UTC m=+51.696576492"
Jul 27 17:02:23.843597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.843553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:23.843597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.843602    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:23.844143 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:23.843666    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:23.844143 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:23.843699    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:23.844143 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:23.843727    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.8437105 +0000 UTC m=+54.429072033 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:23.844143 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:23.843745    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:25.84373398 +0000 UTC m=+54.429095514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:23.970204 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:23.970177    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5brb2_286ed58f-538e-4c7e-8799-310708c75eed/dns-node-resolver/0.log"
Jul 27 17:02:24.084702 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:24.084676    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/1.log"
Jul 27 17:02:24.085115 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:24.085100    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/0.log"
Jul 27 17:02:24.085201 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:24.085134    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb" containerID="8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4" exitCode=255
Jul 27 17:02:24.085201 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:24.085162    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" event={"ID":"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb","Type":"ContainerDied","Data":"8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4"}
Jul 27 17:02:24.085310 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:24.085219    2579 scope.go:117] "RemoveContainer" containerID="34a5ee8fc97389734e33cfafd627ac6c1c996bb079baadc647e557412f39e630"
Jul 27 17:02:24.085521 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:24.085504    2579 scope.go:117] "RemoveContainer" containerID="8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4"
Jul 27 17:02:24.085698 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:24.085678    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-d7xqf_openshift-console-operator(5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podUID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb"
Jul 27 17:02:25.091620 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.091592    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/1.log"
Jul 27 17:02:25.091995 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.091930    2579 scope.go:117] "RemoveContainer" containerID="8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4"
Jul 27 17:02:25.092102 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.092082    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-d7xqf_openshift-console-operator(5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podUID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb"
Jul 27 17:02:25.167925 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.167899    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5hgbt_05546d5f-30c5-44f2-bf23-7bb187cc79b5/node-ca/0.log"
Jul 27 17:02:25.858708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.858671    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:25.858908 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.858719    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:25.858908 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.858802    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:25.858908 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.858821    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:25.858908 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.858864    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:29.85885154 +0000 UTC m=+58.444213073 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:25.858908 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.858876    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:29.85887028 +0000 UTC m=+58.444231814 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:25.959360 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.959327    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:25.959558 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:25.959421    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:25.959558 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.959516    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 17:02:25.959558 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.959558    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:33.959545009 +0000 UTC m=+62.544906543 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : configmap references non-existent config key: service-ca.crt
Jul 27 17:02:25.959691 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:25.959578    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:33.959565242 +0000 UTC m=+62.544926775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : secret "router-metrics-certs-default" not found
Jul 27 17:02:26.160946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.160870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c5glj_da55db76-5640-4deb-916f-58b5c50c9f95/serve-healthcheck-canary/0.log"
Jul 27 17:02:26.284552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.284517    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-vn449"]
Jul 27 17:02:26.327796 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.327708    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-vn449"]
Jul 27 17:02:26.327796 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.327787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.331967 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.331937    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 27 17:02:26.332180 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.331937    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 27 17:02:26.332180 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.331988    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rdqck\""
Jul 27 17:02:26.332180 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.331937    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 27 17:02:26.332180 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.332101    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 27 17:02:26.362594 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.362555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvx9q\" (UniqueName: \"kubernetes.io/projected/ef390506-fa27-47b9-be6c-063508776e7c-kube-api-access-qvx9q\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.362761 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.362608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef390506-fa27-47b9-be6c-063508776e7c-signing-cabundle\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.362761 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.362671    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef390506-fa27-47b9-be6c-063508776e7c-signing-key\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.464003 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.463969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvx9q\" (UniqueName: \"kubernetes.io/projected/ef390506-fa27-47b9-be6c-063508776e7c-kube-api-access-qvx9q\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.464110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.464024    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef390506-fa27-47b9-be6c-063508776e7c-signing-cabundle\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.464110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.464059    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef390506-fa27-47b9-be6c-063508776e7c-signing-key\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.464773 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.464756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ef390506-fa27-47b9-be6c-063508776e7c-signing-cabundle\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.466560 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.466538    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ef390506-fa27-47b9-be6c-063508776e7c-signing-key\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.474993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.474968    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvx9q\" (UniqueName: \"kubernetes.io/projected/ef390506-fa27-47b9-be6c-063508776e7c-kube-api-access-qvx9q\") pod \"service-ca-658d7b7c78-vn449\" (UID: \"ef390506-fa27-47b9-be6c-063508776e7c\") " pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.639452 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.639411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-vn449"
Jul 27 17:02:26.786971 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:26.786908    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-vn449"]
Jul 27 17:02:26.790253 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:26.790224    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef390506_fa27_47b9_be6c_063508776e7c.slice/crio-6459039684d5054e06a18fef7ec2df49729ad3e6f8ab69b2bd603fea24e0a12b WatchSource:0}: Error finding container 6459039684d5054e06a18fef7ec2df49729ad3e6f8ab69b2bd603fea24e0a12b: Status 404 returned error can't find the container with id 6459039684d5054e06a18fef7ec2df49729ad3e6f8ab69b2bd603fea24e0a12b
Jul 27 17:02:27.098303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:27.098219    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-vn449" event={"ID":"ef390506-fa27-47b9-be6c-063508776e7c","Type":"ContainerStarted","Data":"6459039684d5054e06a18fef7ec2df49729ad3e6f8ab69b2bd603fea24e0a12b"}
Jul 27 17:02:28.414604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:28.414569    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:28.414996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:28.414626    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:28.414996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:28.414978    2579 scope.go:117] "RemoveContainer" containerID="8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4"
Jul 27 17:02:28.415187 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:28.415162    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-d7xqf_openshift-console-operator(5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podUID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb"
Jul 27 17:02:29.104720 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:29.104680    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-vn449" event={"ID":"ef390506-fa27-47b9-be6c-063508776e7c","Type":"ContainerStarted","Data":"d6b8fdb04edb1170c9bc6c976ad05e657074d9664e848424d81b168fa49f5047"}
Jul 27 17:02:29.121258 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:29.121206    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-vn449" podStartSLOduration=0.99875567 podStartE2EDuration="3.121190423s" totalImagesPullingTime="2.122434753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:29.120456479 +0000 UTC m=+57.705818036" watchObservedRunningTime="2026-07-27 17:02:29.121190423 +0000 UTC m=+57.706552011"
Jul 27 17:02:29.892986 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:29.892958    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:29.893405 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:29.892996    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:29.893405 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:29.893084    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 27 17:02:29.893405 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:29.893098    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:29.893405 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:29.893148    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:37.893130465 +0000 UTC m=+66.478492001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:29.893405 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:29.893163    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:37.893154848 +0000 UTC m=+66.478516382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : configmap "kube-rbac-proxy" not found
Jul 27 17:02:31.010754 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:31.010722    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bd48h"
Jul 27 17:02:34.031167 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:34.031120    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:34.031640 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:34.031210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:34.031640 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:34.031305    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:50.03128568 +0000 UTC m=+78.616647215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : configmap references non-existent config key: service-ca.crt
Jul 27 17:02:34.031640 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:34.031354    2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 27 17:02:34.031640 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:34.031413    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs podName:dc3e5f58-84bb-4406-b871-1dbdea1e2104 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:50.03139674 +0000 UTC m=+78.616758290 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs") pod "router-default-57898f8749-pb7sc" (UID: "dc3e5f58-84bb-4406-b871-1dbdea1e2104") : secret "router-metrics-certs-default" not found
Jul 27 17:02:36.652764 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.652722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:36.653184 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.652838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:02:36.655437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.655417    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 27 17:02:36.655554 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.655418    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 27 17:02:36.663723 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:36.663701    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 27 17:02:36.663819 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:36.663782    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs podName:158b0a3d-bf31-4102-9fd5-be98f076085c nodeName:}" failed. No retries permitted until 2026-07-27 17:03:40.663762142 +0000 UTC m=+129.249123702 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs") pod "network-metrics-daemon-v6bp7" (UID: "158b0a3d-bf31-4102-9fd5-be98f076085c") : secret "metrics-daemon-secret" not found
Jul 27 17:02:36.666165 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.666136    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/467b5b33-2608-4734-8910-d45ba3de5566-original-pull-secret\") pod \"global-pull-secret-syncer-88gv7\" (UID: \"467b5b33-2608-4734-8910-d45ba3de5566\") " pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:36.753671 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.753636    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:36.756499 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.756464    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 27 17:02:36.767682 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.767660    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 27 17:02:36.777589 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.777562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-882lx\" (UniqueName: \"kubernetes.io/projected/ff986bd4-212e-4ef6-aa1c-6ee37386a7b3-kube-api-access-882lx\") pod \"network-check-target-q46rj\" (UID: \"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3\") " pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:36.802070 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.802042    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-88gv7"
Jul 27 17:02:36.817938 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.817915    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c7drs\""
Jul 27 17:02:36.826277 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.826253    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:36.941551 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.941520    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-88gv7"]
Jul 27 17:02:36.944048 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:36.944020    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467b5b33_2608_4734_8910_d45ba3de5566.slice/crio-b22556c8cca795fd2382aa2885a179e7b22d21e404ed60a92a7523d06501d47d WatchSource:0}: Error finding container b22556c8cca795fd2382aa2885a179e7b22d21e404ed60a92a7523d06501d47d: Status 404 returned error can't find the container with id b22556c8cca795fd2382aa2885a179e7b22d21e404ed60a92a7523d06501d47d
Jul 27 17:02:36.959157 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:36.959132    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-q46rj"]
Jul 27 17:02:36.961607 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:36.961584    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff986bd4_212e_4ef6_aa1c_6ee37386a7b3.slice/crio-1b61777bcbdd88ec30d0bfc961f0a53f3e6c067ea2c5d88d4f9d7f1af652c86d WatchSource:0}: Error finding container 1b61777bcbdd88ec30d0bfc961f0a53f3e6c067ea2c5d88d4f9d7f1af652c86d: Status 404 returned error can't find the container with id 1b61777bcbdd88ec30d0bfc961f0a53f3e6c067ea2c5d88d4f9d7f1af652c86d
Jul 27 17:02:37.127950 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.127913    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88gv7" event={"ID":"467b5b33-2608-4734-8910-d45ba3de5566","Type":"ContainerStarted","Data":"b22556c8cca795fd2382aa2885a179e7b22d21e404ed60a92a7523d06501d47d"}
Jul 27 17:02:37.128981 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.128960    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q46rj" event={"ID":"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3","Type":"ContainerStarted","Data":"1b61777bcbdd88ec30d0bfc961f0a53f3e6c067ea2c5d88d4f9d7f1af652c86d"}
Jul 27 17:02:37.761825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.761785    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:37.764500 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.764444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d51131f9-ae3d-48ce-94b7-ce04477ef8f1-metrics-tls\") pod \"dns-default-tzvrv\" (UID: \"d51131f9-ae3d-48ce-94b7-ce04477ef8f1\") " pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:37.963380 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.963344    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:37.963562 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.963405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:37.963635 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:37.963581    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:37.963687 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:37.963658    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls podName:a8fa5de4-86ff-43ff-923e-5a3bf987c540 nodeName:}" failed. No retries permitted until 2026-07-27 17:02:53.963638597 +0000 UTC m=+82.549000133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mz5cv" (UID: "a8fa5de4-86ff-43ff-923e-5a3bf987c540") : secret "insights-runtime-extractor-tls" not found
Jul 27 17:02:37.963981 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.963954    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a8fa5de4-86ff-43ff-923e-5a3bf987c540-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:37.965673 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.965655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5sbpw\""
Jul 27 17:02:37.973623 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:37.973604    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:38.120399 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:38.120295    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tzvrv"]
Jul 27 17:02:38.122958 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:38.122927    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51131f9_ae3d_48ce_94b7_ce04477ef8f1.slice/crio-161504e251c60984adc85981b6a4616a48c0a85c382edae43d1dd4a91140ee84 WatchSource:0}: Error finding container 161504e251c60984adc85981b6a4616a48c0a85c382edae43d1dd4a91140ee84: Status 404 returned error can't find the container with id 161504e251c60984adc85981b6a4616a48c0a85c382edae43d1dd4a91140ee84
Jul 27 17:02:38.133562 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:38.133536    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tzvrv" event={"ID":"d51131f9-ae3d-48ce-94b7-ce04477ef8f1","Type":"ContainerStarted","Data":"161504e251c60984adc85981b6a4616a48c0a85c382edae43d1dd4a91140ee84"}
Jul 27 17:02:43.153813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.153763    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tzvrv" event={"ID":"d51131f9-ae3d-48ce-94b7-ce04477ef8f1","Type":"ContainerStarted","Data":"847f0644be53d65ddb7aa863e8c3f96c7318e1f4dd00f53804aa1ba5c8df0d1e"}
Jul 27 17:02:43.154288 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.153915    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:43.154288 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.153957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tzvrv" event={"ID":"d51131f9-ae3d-48ce-94b7-ce04477ef8f1","Type":"ContainerStarted","Data":"05cfe28e3ab592bf047cb40deddb9fcffe5db03dc7a490fbb5641327d353f41b"}
Jul 27 17:02:43.155182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.155161    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-q46rj" event={"ID":"ff986bd4-212e-4ef6-aa1c-6ee37386a7b3","Type":"ContainerStarted","Data":"a134582c85082d322060c863c9c02103ce08ed6c0efe4f9ff4ce87abc823126b"}
Jul 27 17:02:43.155276 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.155212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:02:43.156516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.156469    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-88gv7" event={"ID":"467b5b33-2608-4734-8910-d45ba3de5566","Type":"ContainerStarted","Data":"f881b9e8726055ab41a183b1a1c2e7f9139413eefd4dd81031173cb5eb91fb38"}
Jul 27 17:02:43.171265 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.171226    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tzvrv" podStartSLOduration=33.842253675 podStartE2EDuration="38.171215392s" totalImagesPullingTime="4.328961717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:43.170966634 +0000 UTC m=+71.756328189" watchObservedRunningTime="2026-07-27 17:02:43.171215392 +0000 UTC m=+71.756576948"
Jul 27 17:02:43.184776 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.184736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-88gv7" podStartSLOduration=65.66172035 podStartE2EDuration="1m11.184724807s" totalImagesPullingTime="5.523004457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:43.184494087 +0000 UTC m=+71.769855830" watchObservedRunningTime="2026-07-27 17:02:43.184724807 +0000 UTC m=+71.770086362"
Jul 27 17:02:43.888055 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:43.888025    2579 scope.go:117] "RemoveContainer" containerID="8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4"
Jul 27 17:02:44.161519 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.161424    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:02:44.161875 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.161831    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/1.log"
Jul 27 17:02:44.161875 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.161864    2579 generic.go:356] "Generic (PLEG): container finished" podID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb" containerID="fb2f63086f969efc965e60adf508d8a75bcbc02d13f8cfcb217fe8a3ccba40b1" exitCode=255
Jul 27 17:02:44.161997 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.161974    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" event={"ID":"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb","Type":"ContainerDied","Data":"fb2f63086f969efc965e60adf508d8a75bcbc02d13f8cfcb217fe8a3ccba40b1"}
Jul 27 17:02:44.162038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.162028    2579 scope.go:117] "RemoveContainer" containerID="8f534a21b6f8467158fd69cbeb1431a1b77f34953048f1d11038085ab6e3e4a4"
Jul 27 17:02:44.162363 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.162337    2579 scope.go:117] "RemoveContainer" containerID="fb2f63086f969efc965e60adf508d8a75bcbc02d13f8cfcb217fe8a3ccba40b1"
Jul 27 17:02:44.162554 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:44.162533    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-d7xqf_openshift-console-operator(5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podUID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb"
Jul 27 17:02:44.180201 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:44.180162    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-q46rj" podStartSLOduration=66.692393487 podStartE2EDuration="1m12.180148177s" totalImagesPullingTime="5.48775469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:43.198579152 +0000 UTC m=+71.783940715" watchObservedRunningTime="2026-07-27 17:02:44.180148177 +0000 UTC m=+72.765509757"
Jul 27 17:02:45.166838 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:45.166810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:02:47.499752 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.499717    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"]
Jul 27 17:02:47.504029 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.504007    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"]
Jul 27 17:02:47.504164 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.504130    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.507378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.507357    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.508610    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.508892    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-d246s\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.509097    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.510367    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.510832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.510945    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.511235    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.511542    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 27 17:02:47.511763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.511546    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 27 17:02:47.512184 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.511878    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 27 17:02:47.525101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.525080    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"]
Jul 27 17:02:47.526027 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.526005    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"]
Jul 27 17:02:47.639777 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.639745    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.639944 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.639797    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-ca\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.639944 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.639846    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e347bd5f-7056-4bd8-a43e-a651231f614b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.639944 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.639866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlg7g\" (UniqueName: \"kubernetes.io/projected/e347bd5f-7056-4bd8-a43e-a651231f614b-kube-api-access-tlg7g\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.640085 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.639937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.640085 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.639972    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/364f2e29-1e5d-4766-a7bb-4bd6be3a9483-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-nw4hs\" (UID: \"364f2e29-1e5d-4766-a7bb-4bd6be3a9483\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.640085 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.640015    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-hub\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.640085 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.640067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364f2e29-1e5d-4766-a7bb-4bd6be3a9483-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nw4hs\" (UID: \"364f2e29-1e5d-4766-a7bb-4bd6be3a9483\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.740582 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/364f2e29-1e5d-4766-a7bb-4bd6be3a9483-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-nw4hs\" (UID: \"364f2e29-1e5d-4766-a7bb-4bd6be3a9483\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.740794 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740605    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-hub\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.740794 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740720    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364f2e29-1e5d-4766-a7bb-4bd6be3a9483-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nw4hs\" (UID: \"364f2e29-1e5d-4766-a7bb-4bd6be3a9483\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.740794 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.740967 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740797    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-ca\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.740967 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740841    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e347bd5f-7056-4bd8-a43e-a651231f614b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.740967 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740858    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlg7g\" (UniqueName: \"kubernetes.io/projected/e347bd5f-7056-4bd8-a43e-a651231f614b-kube-api-access-tlg7g\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.740967 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.740907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.741360 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.741334    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/364f2e29-1e5d-4766-a7bb-4bd6be3a9483-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-nw4hs\" (UID: \"364f2e29-1e5d-4766-a7bb-4bd6be3a9483\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.741622 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.741589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e347bd5f-7056-4bd8-a43e-a651231f614b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.743704 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.743679    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-hub\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.743969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.743953    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/364f2e29-1e5d-4766-a7bb-4bd6be3a9483-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nw4hs\" (UID: \"364f2e29-1e5d-4766-a7bb-4bd6be3a9483\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.744016 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.743961    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.744057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.744040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-ca\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.744158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.744133    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e347bd5f-7056-4bd8-a43e-a651231f614b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.749182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.749159    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlg7g\" (UniqueName: \"kubernetes.io/projected/e347bd5f-7056-4bd8-a43e-a651231f614b-kube-api-access-tlg7g\") pod \"cluster-proxy-proxy-agent-5787d59457-p76cn\" (UID: \"e347bd5f-7056-4bd8-a43e-a651231f614b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.817008 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.816937    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"
Jul 27 17:02:47.834166 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.834144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"
Jul 27 17:02:47.953938 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.953916    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs"]
Jul 27 17:02:47.958214 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:47.957753    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364f2e29_1e5d_4766_a7bb_4bd6be3a9483.slice/crio-3ff709b86f0d10aa255095ced91d0513bc79fe25966464243a124abc1ad9da20 WatchSource:0}: Error finding container 3ff709b86f0d10aa255095ced91d0513bc79fe25966464243a124abc1ad9da20: Status 404 returned error can't find the container with id 3ff709b86f0d10aa255095ced91d0513bc79fe25966464243a124abc1ad9da20
Jul 27 17:02:47.973026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:47.973001    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn"]
Jul 27 17:02:47.975225 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:47.975206    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode347bd5f_7056_4bd8_a43e_a651231f614b.slice/crio-d467bff1834dfc0d06e725ff0ef0da29e8fb462cdb2335c65f63446ad7602bfa WatchSource:0}: Error finding container d467bff1834dfc0d06e725ff0ef0da29e8fb462cdb2335c65f63446ad7602bfa: Status 404 returned error can't find the container with id d467bff1834dfc0d06e725ff0ef0da29e8fb462cdb2335c65f63446ad7602bfa
Jul 27 17:02:48.177137 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:48.177039    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs" event={"ID":"364f2e29-1e5d-4766-a7bb-4bd6be3a9483","Type":"ContainerStarted","Data":"3ff709b86f0d10aa255095ced91d0513bc79fe25966464243a124abc1ad9da20"}
Jul 27 17:02:48.178062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:48.178040    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn" event={"ID":"e347bd5f-7056-4bd8-a43e-a651231f614b","Type":"ContainerStarted","Data":"d467bff1834dfc0d06e725ff0ef0da29e8fb462cdb2335c65f63446ad7602bfa"}
Jul 27 17:02:48.415219 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:48.415179    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:48.415375 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:48.415244    2579 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:02:48.415640 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:48.415624    2579 scope.go:117] "RemoveContainer" containerID="fb2f63086f969efc965e60adf508d8a75bcbc02d13f8cfcb217fe8a3ccba40b1"
Jul 27 17:02:48.415810 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:48.415792    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-d7xqf_openshift-console-operator(5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podUID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb"
Jul 27 17:02:50.059267 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.059236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:50.059650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.059329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:50.059904 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.059883    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc3e5f58-84bb-4406-b871-1dbdea1e2104-service-ca-bundle\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:50.061820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.061762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc3e5f58-84bb-4406-b871-1dbdea1e2104-metrics-certs\") pod \"router-default-57898f8749-pb7sc\" (UID: \"dc3e5f58-84bb-4406-b871-1dbdea1e2104\") " pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:50.282158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.282113    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7zvk7\""
Jul 27 17:02:50.290374 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.290306    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:50.451199 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:50.451162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-57898f8749-pb7sc"]
Jul 27 17:02:50.452790 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:50.452758    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3e5f58_84bb_4406_b871_1dbdea1e2104.slice/crio-52506c0d16bcef90d4ae795a4a11883fa7f034ee41451b97b04712d7b34468fe WatchSource:0}: Error finding container 52506c0d16bcef90d4ae795a4a11883fa7f034ee41451b97b04712d7b34468fe: Status 404 returned error can't find the container with id 52506c0d16bcef90d4ae795a4a11883fa7f034ee41451b97b04712d7b34468fe
Jul 27 17:02:51.195699 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.195658    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs" event={"ID":"364f2e29-1e5d-4766-a7bb-4bd6be3a9483","Type":"ContainerStarted","Data":"5e427f3ea840267734eb70be497fd8b99cea2c8cbf6b508b34f34b9409bda89b"}
Jul 27 17:02:51.197856 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.197769    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57898f8749-pb7sc" event={"ID":"dc3e5f58-84bb-4406-b871-1dbdea1e2104","Type":"ContainerStarted","Data":"aa938ba561958c221985005a710bc295fb517d471fbf7d994213122ad7f7f317"}
Jul 27 17:02:51.197856 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.197817    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-57898f8749-pb7sc" event={"ID":"dc3e5f58-84bb-4406-b871-1dbdea1e2104","Type":"ContainerStarted","Data":"52506c0d16bcef90d4ae795a4a11883fa7f034ee41451b97b04712d7b34468fe"}
Jul 27 17:02:51.226270 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.225704    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nw4hs" podStartSLOduration=2.06234347 podStartE2EDuration="4.225686115s" totalImagesPullingTime="2.163342645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:51.224627368 +0000 UTC m=+79.809988924" watchObservedRunningTime="2026-07-27 17:02:51.225686115 +0000 UTC m=+79.811047672"
Jul 27 17:02:51.245513 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.245443    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-57898f8749-pb7sc" podStartSLOduration=33.245425171 podStartE2EDuration="33.245425171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:51.243839808 +0000 UTC m=+79.829201365" watchObservedRunningTime="2026-07-27 17:02:51.245425171 +0000 UTC m=+79.830786729"
Jul 27 17:02:51.290554 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.290525    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:51.293746 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:51.293720    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:52.202632 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:52.202595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:52.203915 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:52.203890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-57898f8749-pb7sc"
Jul 27 17:02:53.164244 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.164220    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tzvrv"
Jul 27 17:02:53.207957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.207598    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn" event={"ID":"e347bd5f-7056-4bd8-a43e-a651231f614b","Type":"ContainerStarted","Data":"ebcfbe2c16067777559c5cc23145ee76e50f4282dc88d681becd1c35a19cfc9f"}
Jul 27 17:02:53.207957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.207653    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn" event={"ID":"e347bd5f-7056-4bd8-a43e-a651231f614b","Type":"ContainerStarted","Data":"19173d30b482db57e18a872402a7aa823ba36bd6aef587566da3942cee1c3770"}
Jul 27 17:02:53.207957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.207676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn" event={"ID":"e347bd5f-7056-4bd8-a43e-a651231f614b","Type":"ContainerStarted","Data":"2091f1a17d6bb3e973929c67ea2168a69059966166b5522ab0533e6077e83800"}
Jul 27 17:02:53.228836 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.228609    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5787d59457-p76cn" podStartSLOduration=1.19157054 podStartE2EDuration="6.228589566s" totalImagesPullingTime="5.037019026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:02:53.227571488 +0000 UTC m=+81.812933048" watchObservedRunningTime="2026-07-27 17:02:53.228589566 +0000 UTC m=+81.813951123"
Jul 27 17:02:53.995648 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.995606    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:53.998092 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:53.998054    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a8fa5de4-86ff-43ff-923e-5a3bf987c540-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mz5cv\" (UID: \"a8fa5de4-86ff-43ff-923e-5a3bf987c540\") " pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:54.007038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:54.007015    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hn8cb\""
Jul 27 17:02:54.014812 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:54.014796    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mz5cv"
Jul 27 17:02:54.147062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:54.147040    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mz5cv"]
Jul 27 17:02:54.149171 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:02:54.149141    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8fa5de4_86ff_43ff_923e_5a3bf987c540.slice/crio-98788a7b0e29636c492db2c2c20e50ab523de885d15d54dd869cdd01ba7f1f8e WatchSource:0}: Error finding container 98788a7b0e29636c492db2c2c20e50ab523de885d15d54dd869cdd01ba7f1f8e: Status 404 returned error can't find the container with id 98788a7b0e29636c492db2c2c20e50ab523de885d15d54dd869cdd01ba7f1f8e
Jul 27 17:02:54.211775 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:54.211740    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mz5cv" event={"ID":"a8fa5de4-86ff-43ff-923e-5a3bf987c540","Type":"ContainerStarted","Data":"98788a7b0e29636c492db2c2c20e50ab523de885d15d54dd869cdd01ba7f1f8e"}
Jul 27 17:02:59.229383 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:59.229295    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mz5cv" event={"ID":"a8fa5de4-86ff-43ff-923e-5a3bf987c540","Type":"ContainerStarted","Data":"70fe13ebbe77a91e28320c73c736464a9392f6ee7dd04d3befbefadf230c8bb4"}
Jul 27 17:02:59.888255 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:02:59.888224    2579 scope.go:117] "RemoveContainer" containerID="fb2f63086f969efc965e60adf508d8a75bcbc02d13f8cfcb217fe8a3ccba40b1"
Jul 27 17:02:59.888502 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:02:59.888433    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-d7xqf_openshift-console-operator(5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb)\"" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podUID="5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb"
Jul 27 17:03:03.246027 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:03.245984    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mz5cv" event={"ID":"a8fa5de4-86ff-43ff-923e-5a3bf987c540","Type":"ContainerStarted","Data":"454393e396c48c0b18dafbfa7547630c0830cee2842ed2019aa64a739cc59d0b"}
Jul 27 17:03:05.954036 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.954003    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ngjnn"]
Jul 27 17:03:05.973829 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.973794    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:05.976152 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.976128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 27 17:03:05.976283 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.976231    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 27 17:03:05.976820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.976447    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 27 17:03:05.976820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.976566    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 27 17:03:05.976820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.976679    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 27 17:03:05.977386 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.977370    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m4bj4\""
Jul 27 17:03:05.977726 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:05.977384    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 27 17:03:06.095077 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095055    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-textfile\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095187 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-sys\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095187 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095124    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9vt\" (UniqueName: \"kubernetes.io/projected/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-kube-api-access-9m9vt\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095187 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095151    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-wtmp\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095187 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095184    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095217    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-metrics-client-ca\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095234    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-root\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095257    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-tls\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.095391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.095293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196529 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196496    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-sys\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196700 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196555    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9vt\" (UniqueName: \"kubernetes.io/projected/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-kube-api-access-9m9vt\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196700 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196557    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-sys\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196700 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-wtmp\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196700 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196617    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196700 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196665    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-metrics-client-ca\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196700 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-root\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196713    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-tls\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196744    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.196987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.196796    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-textfile\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.197687 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.197192    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-textfile\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.197687 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.197259    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-wtmp\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.197687 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.197266    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-root\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.197687 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:03:06.197348    2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 27 17:03:06.197687 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:03:06.197401    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-tls podName:b5d7feb3-e6f9-490e-911e-dc9af38fd78a nodeName:}" failed. No retries permitted until 2026-07-27 17:03:06.697382361 +0000 UTC m=+95.282743901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-tls") pod "node-exporter-ngjnn" (UID: "b5d7feb3-e6f9-490e-911e-dc9af38fd78a") : secret "node-exporter-tls" not found
Jul 27 17:03:06.197687 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.197584    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-metrics-client-ca\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.198072 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.197739    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.200166 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.200142    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.211317 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.211263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9vt\" (UniqueName: \"kubernetes.io/projected/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-kube-api-access-9m9vt\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.259488 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.259450    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mz5cv" event={"ID":"a8fa5de4-86ff-43ff-923e-5a3bf987c540","Type":"ContainerStarted","Data":"63ff3226a86aaf28bd9dec6522d3d5cdc676f96a556a988306fb642b28872185"}
Jul 27 17:03:06.281055 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.281000    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mz5cv" podStartSLOduration=32.504432026 podStartE2EDuration="44.280980893s" totalImagesPullingTime="11.776548867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 17:02:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:06.280881582 +0000 UTC m=+94.866243139" watchObservedRunningTime="2026-07-27 17:03:06.280980893 +0000 UTC m=+94.866342451"
Jul 27 17:03:06.700463 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.700378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-tls\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.702915 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.702891    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b5d7feb3-e6f9-490e-911e-dc9af38fd78a-node-exporter-tls\") pod \"node-exporter-ngjnn\" (UID: \"b5d7feb3-e6f9-490e-911e-dc9af38fd78a\") " pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.885126 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:06.885087    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ngjnn"
Jul 27 17:03:06.896332 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:03:06.896294    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d7feb3_e6f9_490e_911e_dc9af38fd78a.slice/crio-7039da54c9d4874415dbb4c54a73d063822471aaf927a7a0c0cfbeeeb66fe5a5 WatchSource:0}: Error finding container 7039da54c9d4874415dbb4c54a73d063822471aaf927a7a0c0cfbeeeb66fe5a5: Status 404 returned error can't find the container with id 7039da54c9d4874415dbb4c54a73d063822471aaf927a7a0c0cfbeeeb66fe5a5
Jul 27 17:03:07.264443 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:07.264408    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngjnn" event={"ID":"b5d7feb3-e6f9-490e-911e-dc9af38fd78a","Type":"ContainerStarted","Data":"7039da54c9d4874415dbb4c54a73d063822471aaf927a7a0c0cfbeeeb66fe5a5"}
Jul 27 17:03:10.276812 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:10.276772    2579 generic.go:356] "Generic (PLEG): container finished" podID="b5d7feb3-e6f9-490e-911e-dc9af38fd78a" containerID="f0b379189e09f7369242e9d8c18302c9b6c80794a712296e6d720d87e4040b1e" exitCode=0
Jul 27 17:03:10.277191 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:10.276873    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngjnn" event={"ID":"b5d7feb3-e6f9-490e-911e-dc9af38fd78a","Type":"ContainerDied","Data":"f0b379189e09f7369242e9d8c18302c9b6c80794a712296e6d720d87e4040b1e"}
Jul 27 17:03:11.282636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:11.282593    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngjnn" event={"ID":"b5d7feb3-e6f9-490e-911e-dc9af38fd78a","Type":"ContainerStarted","Data":"997721810a65a0e41cdd2d84ad13c36537ef972f8b99e1b05ae8a6417d2c7920"}
Jul 27 17:03:11.282990 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:11.282660    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngjnn" event={"ID":"b5d7feb3-e6f9-490e-911e-dc9af38fd78a","Type":"ContainerStarted","Data":"388d4087a57e18a41d497a0d42092006f84cdf200d2ac28d006b9eecda9d4968"}
Jul 27 17:03:11.304716 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:11.304669    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ngjnn" podStartSLOduration=3.522306747 podStartE2EDuration="6.304656298s" totalImagesPullingTime="2.711319274s" totalInitContainerRuntime="71.030277ms" isStatefulPod=true podCreationTimestamp="2026-07-27 17:03:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:11.3031759 +0000 UTC m=+99.888537466" watchObservedRunningTime="2026-07-27 17:03:11.304656298 +0000 UTC m=+99.890017854"
Jul 27 17:03:11.888890 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:11.888864    2579 scope.go:117] "RemoveContainer" containerID="fb2f63086f969efc965e60adf508d8a75bcbc02d13f8cfcb217fe8a3ccba40b1"
Jul 27 17:03:12.287453 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.287425    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:03:12.287916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.287554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" event={"ID":"5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb","Type":"ContainerStarted","Data":"0b0aa716e3bde1722857ff3f1c3cd6862c9807a25ee7386cae7c83245c3ddb70"}
Jul 27 17:03:12.288065 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.288046    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:03:12.292816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.292793    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf"
Jul 27 17:03:12.308570 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.308519    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-d7xqf" podStartSLOduration=50.724654843 podStartE2EDuration="54.308500822s" totalImagesPullingTime="3.583845979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:02:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:12.307393074 +0000 UTC m=+100.892754629" watchObservedRunningTime="2026-07-27 17:03:12.308500822 +0000 UTC m=+100.893862379"
Jul 27 17:03:12.380144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.380105    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-mbhj4"]
Jul 27 17:03:12.383195 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.383170    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:12.385932 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.385911    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 27 17:03:12.386062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.386039    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-qsbgf\""
Jul 27 17:03:12.386177 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.386160    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 27 17:03:12.404837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.404798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-mbhj4"]
Jul 27 17:03:12.548872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.548783    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx9pq\" (UniqueName: \"kubernetes.io/projected/6dbcbb31-5d54-4be2-ac41-cb4595df7c6e-kube-api-access-dx9pq\") pod \"downloads-7c97dc4555-mbhj4\" (UID: \"6dbcbb31-5d54-4be2-ac41-cb4595df7c6e\") " pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:12.650178 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.650148    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx9pq\" (UniqueName: \"kubernetes.io/projected/6dbcbb31-5d54-4be2-ac41-cb4595df7c6e-kube-api-access-dx9pq\") pod \"downloads-7c97dc4555-mbhj4\" (UID: \"6dbcbb31-5d54-4be2-ac41-cb4595df7c6e\") " pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:12.663264 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.663236    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx9pq\" (UniqueName: \"kubernetes.io/projected/6dbcbb31-5d54-4be2-ac41-cb4595df7c6e-kube-api-access-dx9pq\") pod \"downloads-7c97dc4555-mbhj4\" (UID: \"6dbcbb31-5d54-4be2-ac41-cb4595df7c6e\") " pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:12.694164 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.694137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:12.822881 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:12.822802    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-mbhj4"]
Jul 27 17:03:12.825544 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:03:12.825520    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbcbb31_5d54_4be2_ac41_cb4595df7c6e.slice/crio-a48b90f2a2764a9155416c3415615480124c796c6b5ea4e74e4979f1b3559e0e WatchSource:0}: Error finding container a48b90f2a2764a9155416c3415615480124c796c6b5ea4e74e4979f1b3559e0e: Status 404 returned error can't find the container with id a48b90f2a2764a9155416c3415615480124c796c6b5ea4e74e4979f1b3559e0e
Jul 27 17:03:13.293454 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:13.293418    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-mbhj4" event={"ID":"6dbcbb31-5d54-4be2-ac41-cb4595df7c6e","Type":"ContainerStarted","Data":"a48b90f2a2764a9155416c3415615480124c796c6b5ea4e74e4979f1b3559e0e"}
Jul 27 17:03:14.164861 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:14.164833    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-q46rj"
Jul 27 17:03:29.356698 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:29.356652    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-mbhj4" event={"ID":"6dbcbb31-5d54-4be2-ac41-cb4595df7c6e","Type":"ContainerStarted","Data":"7dff6df1e7e04b4b5b17190a53090a1d789f025b51c7ffb7a057e495e16a6fce"}
Jul 27 17:03:29.357202 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:29.356869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:29.373594 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:29.373560    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-mbhj4"
Jul 27 17:03:29.377876 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:29.377827    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-mbhj4" podStartSLOduration=1.5813767570000001 podStartE2EDuration="17.377786207s" totalImagesPullingTime="15.79640945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:29.377742638 +0000 UTC m=+117.963104198" watchObservedRunningTime="2026-07-27 17:03:29.377786207 +0000 UTC m=+117.963147757"
Jul 27 17:03:31.657412 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.657373    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b6fcc4-wf7wd"]
Jul 27 17:03:31.684827 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.684798    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b6fcc4-wf7wd"]
Jul 27 17:03:31.684991 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.684914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.687782 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.687756    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:03:31.688743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.688716    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:03:31.688893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.688770    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:03:31.688893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.688844    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:03:31.689026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.688896    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:03:31.689026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.688940    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bg6n7\""
Jul 27 17:03:31.693331 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.693307    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:03:31.810542 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-oauth-config\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.810693 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-serving-cert\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.810693 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810582    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-service-ca\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.810693 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjzsb\" (UniqueName: \"kubernetes.io/projected/d7b0b05f-5b23-4674-8e31-3bf8f4403500-kube-api-access-jjzsb\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.810693 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810685    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-oauth-serving-cert\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.810897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-config\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.810897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.810762    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-trusted-ca-bundle\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912069 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.911984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-oauth-config\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912069 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.912020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-serving-cert\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912069 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.912041    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-service-ca\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912069 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.912068    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjzsb\" (UniqueName: \"kubernetes.io/projected/d7b0b05f-5b23-4674-8e31-3bf8f4403500-kube-api-access-jjzsb\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912383 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.912098    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-oauth-serving-cert\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912383 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.912141    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-config\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.912383 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.912174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-trusted-ca-bundle\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.914943 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.914919    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:03:31.915079 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.915009    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:03:31.915217 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.915198    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:03:31.915298 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.915253    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:03:31.915829 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.915809    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:03:31.923010 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.922967    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-oauth-serving-cert\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.923127 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.922989    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-config\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.923607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.923585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-service-ca\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.927045 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.927000    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-serving-cert\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.927294 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.927273    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:03:31.927689 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.927668    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjzsb\" (UniqueName: \"kubernetes.io/projected/d7b0b05f-5b23-4674-8e31-3bf8f4403500-kube-api-access-jjzsb\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.927837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.927817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-oauth-config\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.933113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.933087    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-trusted-ca-bundle\") pod \"console-9b6fcc4-wf7wd\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") " pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:31.999616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:31.999574    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bg6n7\""
Jul 27 17:03:32.007743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:32.007712    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:32.192955 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:32.192883    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b6fcc4-wf7wd"]
Jul 27 17:03:32.196342 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:03:32.196310    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b0b05f_5b23_4674_8e31_3bf8f4403500.slice/crio-651f0e2f5bec6bce2f22463c107cdf6c962812a24418f8c8622ee9cbe8f4ee2d WatchSource:0}: Error finding container 651f0e2f5bec6bce2f22463c107cdf6c962812a24418f8c8622ee9cbe8f4ee2d: Status 404 returned error can't find the container with id 651f0e2f5bec6bce2f22463c107cdf6c962812a24418f8c8622ee9cbe8f4ee2d
Jul 27 17:03:32.369559 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:32.369520    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b6fcc4-wf7wd" event={"ID":"d7b0b05f-5b23-4674-8e31-3bf8f4403500","Type":"ContainerStarted","Data":"651f0e2f5bec6bce2f22463c107cdf6c962812a24418f8c8622ee9cbe8f4ee2d"}
Jul 27 17:03:36.388636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:36.388587    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b6fcc4-wf7wd" event={"ID":"d7b0b05f-5b23-4674-8e31-3bf8f4403500","Type":"ContainerStarted","Data":"f304cee5db3eb5968702c357f792e5917094d69137817a56217dae14278f46e1"}
Jul 27 17:03:36.407424 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:36.407377    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9b6fcc4-wf7wd" podStartSLOduration=1.6559794 podStartE2EDuration="5.40736155s" totalImagesPullingTime="3.75138215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:03:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:36.406072276 +0000 UTC m=+124.991433832" watchObservedRunningTime="2026-07-27 17:03:36.40736155 +0000 UTC m=+124.992723106"
Jul 27 17:03:40.697364 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:40.697323    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:03:40.699654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:40.699629    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/158b0a3d-bf31-4102-9fd5-be98f076085c-metrics-certs\") pod \"network-metrics-daemon-v6bp7\" (UID: \"158b0a3d-bf31-4102-9fd5-be98f076085c\") " pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:03:40.711316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:40.711293    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mxtqx\""
Jul 27 17:03:40.719285 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:40.719270    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-v6bp7"
Jul 27 17:03:40.912494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:40.912445    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-v6bp7"]
Jul 27 17:03:40.915519 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:03:40.915488    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158b0a3d_bf31_4102_9fd5_be98f076085c.slice/crio-55f4d89d75ef6a9509fbfd9eebc2eb85003d358a990bbe05264192822427072c WatchSource:0}: Error finding container 55f4d89d75ef6a9509fbfd9eebc2eb85003d358a990bbe05264192822427072c: Status 404 returned error can't find the container with id 55f4d89d75ef6a9509fbfd9eebc2eb85003d358a990bbe05264192822427072c
Jul 27 17:03:41.409024 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:41.408989    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6bp7" event={"ID":"158b0a3d-bf31-4102-9fd5-be98f076085c","Type":"ContainerStarted","Data":"55f4d89d75ef6a9509fbfd9eebc2eb85003d358a990bbe05264192822427072c"}
Jul 27 17:03:42.008251 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:42.008212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:42.008834 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:42.008273    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:42.013039 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:42.013018    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:42.417186 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:42.417112    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:03:45.423767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:45.423689    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6bp7" event={"ID":"158b0a3d-bf31-4102-9fd5-be98f076085c","Type":"ContainerStarted","Data":"1e757a9f7cff5aee5acaebc8579f33db4296658b2701f4cd492b4e16b95f619c"}
Jul 27 17:03:45.423767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:45.423742    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-v6bp7" event={"ID":"158b0a3d-bf31-4102-9fd5-be98f076085c","Type":"ContainerStarted","Data":"4ce73b1522928a02f38913e0e596617ac339b9a16c9eb8eddb06b386309871e4"}
Jul 27 17:03:45.440137 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:03:45.440089    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-v6bp7" podStartSLOduration=129.571705565 podStartE2EDuration="2m13.440075276s" totalImagesPullingTime="3.868369711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:01:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:03:45.439454393 +0000 UTC m=+134.024815948" watchObservedRunningTime="2026-07-27 17:03:45.440075276 +0000 UTC m=+134.025436831"
Jul 27 17:05:22.596866 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:22.596834    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b6fcc4-wf7wd"]
Jul 27 17:05:47.617956 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.617910    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9b6fcc4-wf7wd" podUID="d7b0b05f-5b23-4674-8e31-3bf8f4403500" containerName="console" containerID="cri-o://f304cee5db3eb5968702c357f792e5917094d69137817a56217dae14278f46e1" gracePeriod=15
Jul 27 17:05:47.837831 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.837794    2579 generic.go:356] "Generic (PLEG): container finished" podID="d7b0b05f-5b23-4674-8e31-3bf8f4403500" containerID="f304cee5db3eb5968702c357f792e5917094d69137817a56217dae14278f46e1" exitCode=0
Jul 27 17:05:47.837992 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.837857    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b6fcc4-wf7wd" event={"ID":"d7b0b05f-5b23-4674-8e31-3bf8f4403500","Type":"ContainerDied","Data":"f304cee5db3eb5968702c357f792e5917094d69137817a56217dae14278f46e1"}
Jul 27 17:05:47.858483 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.858431    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:05:47.937238 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937162    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-serving-cert\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937238 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937201    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjzsb\" (UniqueName: \"kubernetes.io/projected/d7b0b05f-5b23-4674-8e31-3bf8f4403500-kube-api-access-jjzsb\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937238 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-service-ca\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937238 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937238    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-config\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937262    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-trusted-ca-bundle\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-oauth-config\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937581 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937336    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-oauth-serving-cert\") pod \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\" (UID: \"d7b0b05f-5b23-4674-8e31-3bf8f4403500\") "
Jul 27 17:05:47.937785 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937753    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-service-ca" (OuterVolumeSpecName: "service-ca") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:05:47.937849 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937783    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-config" (OuterVolumeSpecName: "console-config") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:05:47.937849 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937814    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:05:47.937849 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.937803    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:05:47.939642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.939618    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:05:47.939642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.939627    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b0b05f-5b23-4674-8e31-3bf8f4403500-kube-api-access-jjzsb" (OuterVolumeSpecName: "kube-api-access-jjzsb") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "kube-api-access-jjzsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:05:47.939767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:47.939681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d7b0b05f-5b23-4674-8e31-3bf8f4403500" (UID: "d7b0b05f-5b23-4674-8e31-3bf8f4403500"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:05:48.038601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038560    2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-oauth-serving-cert\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.038601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038595    2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-serving-cert\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.038601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038604    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjzsb\" (UniqueName: \"kubernetes.io/projected/d7b0b05f-5b23-4674-8e31-3bf8f4403500-kube-api-access-jjzsb\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.038601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038615    2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-service-ca\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.038854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038623    2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-config\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.038854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038631    2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d7b0b05f-5b23-4674-8e31-3bf8f4403500-trusted-ca-bundle\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.038854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.038638    2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d7b0b05f-5b23-4674-8e31-3bf8f4403500-console-oauth-config\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:05:48.842036 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.842002    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b6fcc4-wf7wd" event={"ID":"d7b0b05f-5b23-4674-8e31-3bf8f4403500","Type":"ContainerDied","Data":"651f0e2f5bec6bce2f22463c107cdf6c962812a24418f8c8622ee9cbe8f4ee2d"}
Jul 27 17:05:48.842430 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.842057    2579 scope.go:117] "RemoveContainer" containerID="f304cee5db3eb5968702c357f792e5917094d69137817a56217dae14278f46e1"
Jul 27 17:05:48.842430 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.842016    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b6fcc4-wf7wd"
Jul 27 17:05:48.862550 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.862523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b6fcc4-wf7wd"]
Jul 27 17:05:48.866697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:48.866678    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9b6fcc4-wf7wd"]
Jul 27 17:05:49.891356 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:05:49.891313    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b0b05f-5b23-4674-8e31-3bf8f4403500" path="/var/lib/kubelet/pods/d7b0b05f-5b23-4674-8e31-3bf8f4403500/volumes"
Jul 27 17:06:18.510548 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.510508    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"]
Jul 27 17:06:18.511097 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.510759    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b0b05f-5b23-4674-8e31-3bf8f4403500" containerName="console"
Jul 27 17:06:18.511097 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.510769    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b0b05f-5b23-4674-8e31-3bf8f4403500" containerName="console"
Jul 27 17:06:18.511097 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.510814    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b0b05f-5b23-4674-8e31-3bf8f4403500" containerName="console"
Jul 27 17:06:18.513647 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.513630    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.516529 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.516501    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 27 17:06:18.516687 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.516533    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 27 17:06:18.516998 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.516978    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6564d\""
Jul 27 17:06:18.517071 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.517041    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 27 17:06:18.517071 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.517050    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 27 17:06:18.517167 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.517108    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 27 17:06:18.524039 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.524013    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"]
Jul 27 17:06:18.557949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.557915    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c7bbb73b-8d94-4869-9d4f-50921f6adb80-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.558117 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.557963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9n5\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-kube-api-access-kf9n5\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.558117 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.557989    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-certificates\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.658598 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.658564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c7bbb73b-8d94-4869-9d4f-50921f6adb80-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.658772 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.658613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9n5\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-kube-api-access-kf9n5\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.658772 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.658745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-certificates\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.658882 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:18.658867    2579 secret.go:281] references non-existent secret key: tls.crt
Jul 27 17:06:18.658917 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:18.658886    2579 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 27 17:06:18.658917 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:18.658903    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-flrt2: references non-existent secret key: tls.crt
Jul 27 17:06:18.658978 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:18.658951    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-certificates podName:c7bbb73b-8d94-4869-9d4f-50921f6adb80 nodeName:}" failed. No retries permitted until 2026-07-27 17:06:19.15893685 +0000 UTC m=+287.744298384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-certificates") pod "keda-metrics-apiserver-55557df8c-flrt2" (UID: "c7bbb73b-8d94-4869-9d4f-50921f6adb80") : references non-existent secret key: tls.crt
Jul 27 17:06:18.658978 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.658971    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c7bbb73b-8d94-4869-9d4f-50921f6adb80-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.667573 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.667556    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9n5\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-kube-api-access-kf9n5\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:18.912058 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.911975    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-w9sj9"]
Jul 27 17:06:18.915024 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.915007    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:18.917469 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.917449    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 27 17:06:18.923034 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.923009    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-w9sj9"]
Jul 27 17:06:18.961927 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.961895    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-certificates\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:18.962101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:18.961952    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jp5\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-kube-api-access-q7jp5\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.062329 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.062292    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-certificates\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.062513 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.062342    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jp5\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-kube-api-access-q7jp5\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.062513 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:19.062439    2579 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 27 17:06:19.062513 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:19.062461    2579 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-w9sj9: secret "keda-admission-webhooks-certs" not found
Jul 27 17:06:19.062645 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:06:19.062537    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-certificates podName:36f42f8a-8524-45d6-89c6-740addb4c65f nodeName:}" failed. No retries permitted until 2026-07-27 17:06:19.562521512 +0000 UTC m=+288.147883063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-certificates") pod "keda-admission-56f9798bb4-w9sj9" (UID: "36f42f8a-8524-45d6-89c6-740addb4c65f") : secret "keda-admission-webhooks-certs" not found
Jul 27 17:06:19.079921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.079889    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jp5\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-kube-api-access-q7jp5\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.163585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.163466    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-certificates\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:19.166095 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.166067    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c7bbb73b-8d94-4869-9d4f-50921f6adb80-certificates\") pod \"keda-metrics-apiserver-55557df8c-flrt2\" (UID: \"c7bbb73b-8d94-4869-9d4f-50921f6adb80\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:19.426798 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.426692    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:19.559209 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.559179    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"]
Jul 27 17:06:19.561709 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:06:19.561682    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bbb73b_8d94_4869_9d4f_50921f6adb80.slice/crio-7c30c53a1b266e16e78a91888974cfaab37a737947813d1ce8e3f638bbf433dc WatchSource:0}: Error finding container 7c30c53a1b266e16e78a91888974cfaab37a737947813d1ce8e3f638bbf433dc: Status 404 returned error can't find the container with id 7c30c53a1b266e16e78a91888974cfaab37a737947813d1ce8e3f638bbf433dc
Jul 27 17:06:19.566663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.566637    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-certificates\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.569188 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.569166    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/36f42f8a-8524-45d6-89c6-740addb4c65f-certificates\") pod \"keda-admission-56f9798bb4-w9sj9\" (UID: \"36f42f8a-8524-45d6-89c6-740addb4c65f\") " pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.826953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.826912    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:19.953305 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.953271    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2" event={"ID":"c7bbb73b-8d94-4869-9d4f-50921f6adb80","Type":"ContainerStarted","Data":"7c30c53a1b266e16e78a91888974cfaab37a737947813d1ce8e3f638bbf433dc"}
Jul 27 17:06:19.971913 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:19.971799    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-w9sj9"]
Jul 27 17:06:19.974443 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:06:19.974406    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36f42f8a_8524_45d6_89c6_740addb4c65f.slice/crio-51e4e0acbd6abd20ac5007f5d4c0a53541a9091db4d463f9b044b94ea367e5c3 WatchSource:0}: Error finding container 51e4e0acbd6abd20ac5007f5d4c0a53541a9091db4d463f9b044b94ea367e5c3: Status 404 returned error can't find the container with id 51e4e0acbd6abd20ac5007f5d4c0a53541a9091db4d463f9b044b94ea367e5c3
Jul 27 17:06:20.959874 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:20.959801    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9" event={"ID":"36f42f8a-8524-45d6-89c6-740addb4c65f","Type":"ContainerStarted","Data":"51e4e0acbd6abd20ac5007f5d4c0a53541a9091db4d463f9b044b94ea367e5c3"}
Jul 27 17:06:23.973099 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:23.973052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2" event={"ID":"c7bbb73b-8d94-4869-9d4f-50921f6adb80","Type":"ContainerStarted","Data":"63ea7ee6a3cb2e90e5d906fed3cb00223b6ccb5ae2b0910c15adea0ea6dd17cf"}
Jul 27 17:06:23.973595 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:23.973178    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:23.974450 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:23.974422    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9" event={"ID":"36f42f8a-8524-45d6-89c6-740addb4c65f","Type":"ContainerStarted","Data":"e494bc813934c4805cf14efecfe04500f3f4e80f7fd351c2f9de0651647847c1"}
Jul 27 17:06:23.997696 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:23.997638    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2" podStartSLOduration=2.174427736 podStartE2EDuration="5.99762019s" totalImagesPullingTime="3.823192454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:23.996219709 +0000 UTC m=+292.581581269" watchObservedRunningTime="2026-07-27 17:06:23.99762019 +0000 UTC m=+292.582981776"
Jul 27 17:06:24.017962 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:24.017910    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9" podStartSLOduration=2.608343617 podStartE2EDuration="6.01789296s" totalImagesPullingTime="3.409549343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:06:24.017152411 +0000 UTC m=+292.602513986" watchObservedRunningTime="2026-07-27 17:06:24.01789296 +0000 UTC m=+292.603254542"
Jul 27 17:06:24.978513 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:24.978459    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:06:31.731836 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:31.731809    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:06:31.735312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:31.735291    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:06:31.746175 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:31.746154    2579 kubelet.go:1648] "Image garbage collection succeeded"
Jul 27 17:06:34.983122 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:34.983089    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-flrt2"
Jul 27 17:06:45.989158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:06:45.989127    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-w9sj9"
Jul 27 17:07:37.603977 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.603904    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-bzpsq"]
Jul 27 17:07:37.607008 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.606991    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.609779 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.609755    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 27 17:07:37.609924 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.609785    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 27 17:07:37.609924 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.609846    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vl2qf\""
Jul 27 17:07:37.610818 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.610803    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 27 17:07:37.615791 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.615771    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bzpsq"]
Jul 27 17:07:37.628052 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.628028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dtz\" (UniqueName: \"kubernetes.io/projected/773b4faa-3217-4539-8a27-5246848c6be4-kube-api-access-q4dtz\") pod \"seaweedfs-64568bcd49-bzpsq\" (UID: \"773b4faa-3217-4539-8a27-5246848c6be4\") " pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.628165 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.628083    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/773b4faa-3217-4539-8a27-5246848c6be4-data\") pod \"seaweedfs-64568bcd49-bzpsq\" (UID: \"773b4faa-3217-4539-8a27-5246848c6be4\") " pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.729045 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.729015    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dtz\" (UniqueName: \"kubernetes.io/projected/773b4faa-3217-4539-8a27-5246848c6be4-kube-api-access-q4dtz\") pod \"seaweedfs-64568bcd49-bzpsq\" (UID: \"773b4faa-3217-4539-8a27-5246848c6be4\") " pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.729218 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.729067    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/773b4faa-3217-4539-8a27-5246848c6be4-data\") pod \"seaweedfs-64568bcd49-bzpsq\" (UID: \"773b4faa-3217-4539-8a27-5246848c6be4\") " pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.729387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.729371    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/773b4faa-3217-4539-8a27-5246848c6be4-data\") pod \"seaweedfs-64568bcd49-bzpsq\" (UID: \"773b4faa-3217-4539-8a27-5246848c6be4\") " pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.737577 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.737558    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dtz\" (UniqueName: \"kubernetes.io/projected/773b4faa-3217-4539-8a27-5246848c6be4-kube-api-access-q4dtz\") pod \"seaweedfs-64568bcd49-bzpsq\" (UID: \"773b4faa-3217-4539-8a27-5246848c6be4\") " pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:37.917419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:37.917317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:38.045087 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:38.045059    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-bzpsq"]
Jul 27 17:07:38.047464 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:07:38.047435    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773b4faa_3217_4539_8a27_5246848c6be4.slice/crio-cbd7d3ce9b8377c52ff9b2c4b0bd1bdd44abd5d19146b87c5c929a1154b86036 WatchSource:0}: Error finding container cbd7d3ce9b8377c52ff9b2c4b0bd1bdd44abd5d19146b87c5c929a1154b86036: Status 404 returned error can't find the container with id cbd7d3ce9b8377c52ff9b2c4b0bd1bdd44abd5d19146b87c5c929a1154b86036
Jul 27 17:07:38.048659 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:38.048640    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:07:38.235388 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:38.235350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bzpsq" event={"ID":"773b4faa-3217-4539-8a27-5246848c6be4","Type":"ContainerStarted","Data":"cbd7d3ce9b8377c52ff9b2c4b0bd1bdd44abd5d19146b87c5c929a1154b86036"}
Jul 27 17:07:41.248158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:41.248077    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-bzpsq" event={"ID":"773b4faa-3217-4539-8a27-5246848c6be4","Type":"ContainerStarted","Data":"16e20e4fcac0d1b1eb8a68ccf3075867e8aac709a518cd6177f6e5c3968fba38"}
Jul 27 17:07:41.248158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:41.248143    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:07:41.263724 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:41.263677    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-bzpsq" podStartSLOduration=1.337735582 podStartE2EDuration="4.263661538s" totalImagesPullingTime="2.925925956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:07:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:07:41.263368535 +0000 UTC m=+369.848730093" watchObservedRunningTime="2026-07-27 17:07:41.263661538 +0000 UTC m=+369.849023094"
Jul 27 17:07:47.253679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:07:47.253627    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-bzpsq"
Jul 27 17:08:26.460309 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.460279    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8496f55b4-wjhbs"]
Jul 27 17:08:26.463560 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.463536    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.467106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.467084    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bg6n7\""
Jul 27 17:08:26.467224 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.467123    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 27 17:08:26.467224 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.467087    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 27 17:08:26.467669 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.467652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 27 17:08:26.467809 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.467707    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 27 17:08:26.467925 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.467722    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 27 17:08:26.471436 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.471418    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 27 17:08:26.481689 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.481658    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8496f55b4-wjhbs"]
Jul 27 17:08:26.565420 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565381    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-console-config\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.565420 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565418    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-oauth-serving-cert\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.565675 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8c8\" (UniqueName: \"kubernetes.io/projected/052e022d-0b92-4f6f-87a7-64dd29328497-kube-api-access-zq8c8\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.565675 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-service-ca\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.565675 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565537    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052e022d-0b92-4f6f-87a7-64dd29328497-console-oauth-config\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.565675 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052e022d-0b92-4f6f-87a7-64dd29328497-console-serving-cert\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.565675 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.565575    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-trusted-ca-bundle\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.665920 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.665886    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-oauth-serving-cert\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.665936    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8c8\" (UniqueName: \"kubernetes.io/projected/052e022d-0b92-4f6f-87a7-64dd29328497-kube-api-access-zq8c8\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.665972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-service-ca\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.666004    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052e022d-0b92-4f6f-87a7-64dd29328497-console-oauth-config\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.666021    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052e022d-0b92-4f6f-87a7-64dd29328497-console-serving-cert\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.666045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-trusted-ca-bundle\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666363 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.666221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-console-config\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666777 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.666746    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-oauth-serving-cert\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.666986 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.666966    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-console-config\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.667064 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.667005    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-service-ca\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.667064 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.667044    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052e022d-0b92-4f6f-87a7-64dd29328497-trusted-ca-bundle\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.668463 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.668445    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052e022d-0b92-4f6f-87a7-64dd29328497-console-oauth-config\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.668642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.668627    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052e022d-0b92-4f6f-87a7-64dd29328497-console-serving-cert\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.674987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.674962    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8c8\" (UniqueName: \"kubernetes.io/projected/052e022d-0b92-4f6f-87a7-64dd29328497-kube-api-access-zq8c8\") pod \"console-8496f55b4-wjhbs\" (UID: \"052e022d-0b92-4f6f-87a7-64dd29328497\") " pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.775024 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.774949    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:26.910285 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:26.910263    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8496f55b4-wjhbs"]
Jul 27 17:08:26.912412 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:08:26.912384    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052e022d_0b92_4f6f_87a7_64dd29328497.slice/crio-8021ba502d1424a3a7793b1460398eb4db63a392fcab98f4971e77770e232f80 WatchSource:0}: Error finding container 8021ba502d1424a3a7793b1460398eb4db63a392fcab98f4971e77770e232f80: Status 404 returned error can't find the container with id 8021ba502d1424a3a7793b1460398eb4db63a392fcab98f4971e77770e232f80
Jul 27 17:08:27.408435 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:27.408397    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8496f55b4-wjhbs" event={"ID":"052e022d-0b92-4f6f-87a7-64dd29328497","Type":"ContainerStarted","Data":"8dcb15a8e5496efbc7f0afa22542c7c5091a097f53aa7597368090694df56c7a"}
Jul 27 17:08:27.408654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:27.408453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8496f55b4-wjhbs" event={"ID":"052e022d-0b92-4f6f-87a7-64dd29328497","Type":"ContainerStarted","Data":"8021ba502d1424a3a7793b1460398eb4db63a392fcab98f4971e77770e232f80"}
Jul 27 17:08:27.425319 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:27.425265    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8496f55b4-wjhbs" podStartSLOduration=1.425251523 podStartE2EDuration="1.425251523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:08:27.424620866 +0000 UTC m=+416.009982416" watchObservedRunningTime="2026-07-27 17:08:27.425251523 +0000 UTC m=+416.010613077"
Jul 27 17:08:36.775802 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:36.775765    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:36.776234 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:36.775843    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:36.780689 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:36.780668    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:08:37.447919 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:08:37.447883    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8496f55b4-wjhbs"
Jul 27 17:09:10.415773 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.415724    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"]
Jul 27 17:09:10.419051 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.419034    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.422075 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.422053    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 27 17:09:10.422198 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.422086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 27 17:09:10.422250 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.422207    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:09:10.422352 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.422333    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w5mg5\""
Jul 27 17:09:10.422720 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.422705    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 27 17:09:10.433216 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.433193    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"]
Jul 27 17:09:10.602805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.602771    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6688e3-515f-47ac-b6c2-d72ebedb5102-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.602805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.602807    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.603017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.602831    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.603017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.602859    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fmg\" (UniqueName: \"kubernetes.io/projected/bd6688e3-515f-47ac-b6c2-d72ebedb5102-kube-api-access-98fmg\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.603017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.602912    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.603017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.602969    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.603017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.603007    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703377 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703377 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703364    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6688e3-515f-47ac-b6c2-d72ebedb5102-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703406    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fmg\" (UniqueName: \"kubernetes.io/projected/bd6688e3-515f-47ac-b6c2-d72ebedb5102-kube-api-access-98fmg\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703445    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.703585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.703463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.704209 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.704185    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.705977 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.705956    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.706067 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.706031    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.706227 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.706208    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd6688e3-515f-47ac-b6c2-d72ebedb5102-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.706381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.706362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.711843 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.711819    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd6688e3-515f-47ac-b6c2-d72ebedb5102-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.712310 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.712291    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fmg\" (UniqueName: \"kubernetes.io/projected/bd6688e3-515f-47ac-b6c2-d72ebedb5102-kube-api-access-98fmg\") pod \"istiod-openshift-gateway-5677ccb49b-hkbqw\" (UID: \"bd6688e3-515f-47ac-b6c2-d72ebedb5102\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.733199 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.733178    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:10.859568 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:10.859542    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"]
Jul 27 17:09:10.862031 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:09:10.862006    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6688e3_515f_47ac_b6c2_d72ebedb5102.slice/crio-05739e23df6b0435d7da4dc32bee652135978f3cb01920c1b8aa43112e9b4067 WatchSource:0}: Error finding container 05739e23df6b0435d7da4dc32bee652135978f3cb01920c1b8aa43112e9b4067: Status 404 returned error can't find the container with id 05739e23df6b0435d7da4dc32bee652135978f3cb01920c1b8aa43112e9b4067
Jul 27 17:09:11.564947 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:11.564912    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw" event={"ID":"bd6688e3-515f-47ac-b6c2-d72ebedb5102","Type":"ContainerStarted","Data":"05739e23df6b0435d7da4dc32bee652135978f3cb01920c1b8aa43112e9b4067"}
Jul 27 17:09:14.476597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:14.476563    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:09:14.476833 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:14.476631    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:09:14.578597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:14.578561    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw" event={"ID":"bd6688e3-515f-47ac-b6c2-d72ebedb5102","Type":"ContainerStarted","Data":"8414d3dc5380c55216f20414ae110d7f76d44fb7679f55fe7640b10719ca67f5"}
Jul 27 17:09:14.578783 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:14.578689    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:14.600713 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:14.600656    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw" podStartSLOduration=0.988256512 podStartE2EDuration="4.600639133s" totalImagesPullingTime="3.612382621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:14.600218538 +0000 UTC m=+463.185580094" watchObservedRunningTime="2026-07-27 17:09:14.600639133 +0000 UTC m=+463.186000688"
Jul 27 17:09:15.584570 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:15.584528    2579 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-hkbqw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 27 17:09:15.585012 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:15.584611    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw" podUID="bd6688e3-515f-47ac-b6c2-d72ebedb5102" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 27 17:09:18.583797 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:18.583767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-hkbqw"
Jul 27 17:09:40.201048 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.201014    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"]
Jul 27 17:09:40.211677 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.211647    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:40.212442 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.212416    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"]
Jul 27 17:09:40.214813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.214788    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-5g5kx\""
Jul 27 17:09:40.214813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.214804    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 27 17:09:40.214982 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.214946    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 27 17:09:40.332363 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.332319    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8cjf\" (UniqueName: \"kubernetes.io/projected/f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c-kube-api-access-t8cjf\") pod \"limitador-operator-controller-manager-774d94476f-6fl6q\" (UID: \"f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:40.433216 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.433175    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8cjf\" (UniqueName: \"kubernetes.io/projected/f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c-kube-api-access-t8cjf\") pod \"limitador-operator-controller-manager-774d94476f-6fl6q\" (UID: \"f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:40.448163 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.448130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8cjf\" (UniqueName: \"kubernetes.io/projected/f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c-kube-api-access-t8cjf\") pod \"limitador-operator-controller-manager-774d94476f-6fl6q\" (UID: \"f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c\") " pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:40.522437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.522402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:40.653880 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.653845    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"]
Jul 27 17:09:40.656095 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:09:40.656070    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f5ae01_0ea0_4fa2_98b7_4428d9782d2c.slice/crio-c53148e5197943442ffe2eb64bba30e2991a23d88d0a4181f4dc0b74d92c5b9e WatchSource:0}: Error finding container c53148e5197943442ffe2eb64bba30e2991a23d88d0a4181f4dc0b74d92c5b9e: Status 404 returned error can't find the container with id c53148e5197943442ffe2eb64bba30e2991a23d88d0a4181f4dc0b74d92c5b9e
Jul 27 17:09:40.676289 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:40.676257    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q" event={"ID":"f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c","Type":"ContainerStarted","Data":"c53148e5197943442ffe2eb64bba30e2991a23d88d0a4181f4dc0b74d92c5b9e"}
Jul 27 17:09:43.689753 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:43.689692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q" event={"ID":"f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c","Type":"ContainerStarted","Data":"579af4996147b65c3a503a9b26bc2f6fdbbe8aaa6e3b493d9a56095532b9eb8a"}
Jul 27 17:09:43.690141 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:43.689805    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:43.709939 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:43.709886    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q" podStartSLOduration=0.86826321 podStartE2EDuration="3.709871339s" totalImagesPullingTime="2.841608129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:43.707839084 +0000 UTC m=+492.293200642" watchObservedRunningTime="2026-07-27 17:09:43.709871339 +0000 UTC m=+492.295232914"
Jul 27 17:09:44.525341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.525305    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"]
Jul 27 17:09:44.528605 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.528583    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.531153 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.531136    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-6nfz4\""
Jul 27 17:09:44.542453 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.542430    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"]
Jul 27 17:09:44.668160 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.668125    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpszc\" (UniqueName: \"kubernetes.io/projected/fcf0be9b-c592-4fc7-beae-c71f8133f959-kube-api-access-jpszc\") pod \"kuadrant-operator-controller-manager-66866f5fcc-fl2ps\" (UID: \"fcf0be9b-c592-4fc7-beae-c71f8133f959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.668330 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.668209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcf0be9b-c592-4fc7-beae-c71f8133f959-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-fl2ps\" (UID: \"fcf0be9b-c592-4fc7-beae-c71f8133f959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.768566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.768525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcf0be9b-c592-4fc7-beae-c71f8133f959-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-fl2ps\" (UID: \"fcf0be9b-c592-4fc7-beae-c71f8133f959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.768566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.768585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpszc\" (UniqueName: \"kubernetes.io/projected/fcf0be9b-c592-4fc7-beae-c71f8133f959-kube-api-access-jpszc\") pod \"kuadrant-operator-controller-manager-66866f5fcc-fl2ps\" (UID: \"fcf0be9b-c592-4fc7-beae-c71f8133f959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.769074 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.768941    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fcf0be9b-c592-4fc7-beae-c71f8133f959-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-fl2ps\" (UID: \"fcf0be9b-c592-4fc7-beae-c71f8133f959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.781506 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.781426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpszc\" (UniqueName: \"kubernetes.io/projected/fcf0be9b-c592-4fc7-beae-c71f8133f959-kube-api-access-jpszc\") pod \"kuadrant-operator-controller-manager-66866f5fcc-fl2ps\" (UID: \"fcf0be9b-c592-4fc7-beae-c71f8133f959\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.838600 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.838564    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:44.981348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:44.981316    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"]
Jul 27 17:09:44.987144 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:09:44.987111    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf0be9b_c592_4fc7_beae_c71f8133f959.slice/crio-aa8b4c7649b19ac69f2c801d13023e6ab9c1fbae9d0c80863008bf4ff6009cb6 WatchSource:0}: Error finding container aa8b4c7649b19ac69f2c801d13023e6ab9c1fbae9d0c80863008bf4ff6009cb6: Status 404 returned error can't find the container with id aa8b4c7649b19ac69f2c801d13023e6ab9c1fbae9d0c80863008bf4ff6009cb6
Jul 27 17:09:45.699140 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:45.699107    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps" event={"ID":"fcf0be9b-c592-4fc7-beae-c71f8133f959","Type":"ContainerStarted","Data":"aa8b4c7649b19ac69f2c801d13023e6ab9c1fbae9d0c80863008bf4ff6009cb6"}
Jul 27 17:09:48.221195 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.221159    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-n5pb8"]
Jul 27 17:09:48.224934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.224906    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:09:48.231099 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.231075    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-2tx66\""
Jul 27 17:09:48.235392 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.235368    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-n5pb8"]
Jul 27 17:09:48.297293 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.297259    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kw9h\" (UniqueName: \"kubernetes.io/projected/73f0dfdd-b806-4dcb-9897-65cf2e003c69-kube-api-access-2kw9h\") pod \"authorino-operator-56fb6f6856-n5pb8\" (UID: \"73f0dfdd-b806-4dcb-9897-65cf2e003c69\") " pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:09:48.398441 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.398395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kw9h\" (UniqueName: \"kubernetes.io/projected/73f0dfdd-b806-4dcb-9897-65cf2e003c69-kube-api-access-2kw9h\") pod \"authorino-operator-56fb6f6856-n5pb8\" (UID: \"73f0dfdd-b806-4dcb-9897-65cf2e003c69\") " pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:09:48.409027 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.408992    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kw9h\" (UniqueName: \"kubernetes.io/projected/73f0dfdd-b806-4dcb-9897-65cf2e003c69-kube-api-access-2kw9h\") pod \"authorino-operator-56fb6f6856-n5pb8\" (UID: \"73f0dfdd-b806-4dcb-9897-65cf2e003c69\") " pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:09:48.539624 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:48.539545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:09:49.038677 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:49.038654    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-56fb6f6856-n5pb8"]
Jul 27 17:09:49.040872 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:09:49.040826    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f0dfdd_b806_4dcb_9897_65cf2e003c69.slice/crio-e4f118dd9ef9496d529a6df8ed690c578720423daa069f031befdefff7725c22 WatchSource:0}: Error finding container e4f118dd9ef9496d529a6df8ed690c578720423daa069f031befdefff7725c22: Status 404 returned error can't find the container with id e4f118dd9ef9496d529a6df8ed690c578720423daa069f031befdefff7725c22
Jul 27 17:09:49.718916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:49.718880    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8" event={"ID":"73f0dfdd-b806-4dcb-9897-65cf2e003c69","Type":"ContainerStarted","Data":"e4f118dd9ef9496d529a6df8ed690c578720423daa069f031befdefff7725c22"}
Jul 27 17:09:49.720407 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:49.720384    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps" event={"ID":"fcf0be9b-c592-4fc7-beae-c71f8133f959","Type":"ContainerStarted","Data":"9519dd0f2f9eb85e2e8da114de56b3d8cf146b0f82ba141d03564ea59a60bfb1"}
Jul 27 17:09:49.720532 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:49.720514    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:09:49.750223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:49.750167    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps" podStartSLOduration=1.7663496539999999 podStartE2EDuration="5.750151445s" totalImagesPullingTime="3.983801791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:49.747717428 +0000 UTC m=+498.333078985" watchObservedRunningTime="2026-07-27 17:09:49.750151445 +0000 UTC m=+498.335513000"
Jul 27 17:09:54.696612 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:54.696582    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-774d94476f-6fl6q"
Jul 27 17:09:56.752735 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:56.752705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8" event={"ID":"73f0dfdd-b806-4dcb-9897-65cf2e003c69","Type":"ContainerStarted","Data":"f556575041d5923e4fd3135266258f8100f942d3e020e6f558b28ff1ca648c53"}
Jul 27 17:09:56.753128 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:56.752810    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:09:56.778361 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:09:56.778312    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8" podStartSLOduration=1.512744539 podStartE2EDuration="8.77829244s" totalImagesPullingTime="7.265547901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:09:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:09:56.777226941 +0000 UTC m=+505.362588500" watchObservedRunningTime="2026-07-27 17:09:56.77829244 +0000 UTC m=+505.363653996"
Jul 27 17:10:00.727142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:00.727114    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-fl2ps"
Jul 27 17:10:07.760210 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:07.760175    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 17:10:32.171803 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.171726    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"]
Jul 27 17:10:32.176509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.176464    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.181852 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.181832    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vpspd\""
Jul 27 17:10:32.182030 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.182010    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 27 17:10:32.189673 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.189655    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"]
Jul 27 17:10:32.220343 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.220318    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"]
Jul 27 17:10:32.231144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.231122    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px568\" (UniqueName: \"kubernetes.io/projected/ad29571a-803f-430a-be96-17b11e401dec-kube-api-access-px568\") pod \"limitador-limitador-85d9db4d5d-cdzw4\" (UID: \"ad29571a-803f-430a-be96-17b11e401dec\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.231258 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.231183    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad29571a-803f-430a-be96-17b11e401dec-config-file\") pod \"limitador-limitador-85d9db4d5d-cdzw4\" (UID: \"ad29571a-803f-430a-be96-17b11e401dec\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.331816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.331779    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad29571a-803f-430a-be96-17b11e401dec-config-file\") pod \"limitador-limitador-85d9db4d5d-cdzw4\" (UID: \"ad29571a-803f-430a-be96-17b11e401dec\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.331816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.331818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px568\" (UniqueName: \"kubernetes.io/projected/ad29571a-803f-430a-be96-17b11e401dec-kube-api-access-px568\") pod \"limitador-limitador-85d9db4d5d-cdzw4\" (UID: \"ad29571a-803f-430a-be96-17b11e401dec\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.332440 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.332421    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad29571a-803f-430a-be96-17b11e401dec-config-file\") pod \"limitador-limitador-85d9db4d5d-cdzw4\" (UID: \"ad29571a-803f-430a-be96-17b11e401dec\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.340206 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.340183    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px568\" (UniqueName: \"kubernetes.io/projected/ad29571a-803f-430a-be96-17b11e401dec-kube-api-access-px568\") pod \"limitador-limitador-85d9db4d5d-cdzw4\" (UID: \"ad29571a-803f-430a-be96-17b11e401dec\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.486163 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.486126    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:32.618144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.618116    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"]
Jul 27 17:10:32.620428 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:10:32.620403    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad29571a_803f_430a_be96_17b11e401dec.slice/crio-76e603f26357281eeec2acf4612c4f48701755195c4cfb1753469dc54e086943 WatchSource:0}: Error finding container 76e603f26357281eeec2acf4612c4f48701755195c4cfb1753469dc54e086943: Status 404 returned error can't find the container with id 76e603f26357281eeec2acf4612c4f48701755195c4cfb1753469dc54e086943
Jul 27 17:10:32.906854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:32.906766    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4" event={"ID":"ad29571a-803f-430a-be96-17b11e401dec","Type":"ContainerStarted","Data":"76e603f26357281eeec2acf4612c4f48701755195c4cfb1753469dc54e086943"}
Jul 27 17:10:34.915458 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:34.915413    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4" event={"ID":"ad29571a-803f-430a-be96-17b11e401dec","Type":"ContainerStarted","Data":"8370dcd40fe1d02ab4099dc9ba39a42237993c4aa5125567a08d78faef9e4c27"}
Jul 27 17:10:34.915845 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:34.915490    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:10:34.931206 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:34.931159    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4" podStartSLOduration=1.679787388 podStartE2EDuration="2.931140738s" totalImagesPullingTime="1.25135335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:10:34.931036704 +0000 UTC m=+543.516398261" watchObservedRunningTime="2026-07-27 17:10:34.931140738 +0000 UTC m=+543.516502309"
Jul 27 17:10:45.920604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:10:45.920574    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-cdzw4"
Jul 27 17:11:31.761163 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:11:31.761135    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:11:31.763059 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:11:31.763040    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:15:41.865041 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.865006    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"]
Jul 27 17:15:41.868750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.868728    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.871378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.871349    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:15:41.871538 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.871376    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"kube-root-ca.crt\""
Jul 27 17:15:41.871538 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.871467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"router-gateway-1-openshift-default-dockercfg-clln6\""
Jul 27 17:15:41.871668 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.871647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-49b5edd9\"/\"openshift-service-ca.crt\""
Jul 27 17:15:41.881737 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.881713    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"]
Jul 27 17:15:41.992074 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992041    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992088    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd376557-6c98-4a3c-a377-319a90face63-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992185    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992220    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992263    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992304    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992522 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992522 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992425    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd376557-6c98-4a3c-a377-319a90face63-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:41.992522 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:41.992456    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lvc\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-kube-api-access-89lvc\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093174    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093190    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd376557-6c98-4a3c-a377-319a90face63-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093217    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lvc\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-kube-api-access-89lvc\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093253    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093284    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd376557-6c98-4a3c-a377-319a90face63-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093832 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093815    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-certs\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.093949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093926    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-credential-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.094172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093955    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-socket\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.094172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.093981    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-data\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.094172 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.094093    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd376557-6c98-4a3c-a377-319a90face63-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.095708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.095685    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-envoy\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.095987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.095970    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd376557-6c98-4a3c-a377-319a90face63-istio-podinfo\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.101652 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.101623    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-istio-token\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.101872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.101828    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lvc\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-kube-api-access-89lvc\") pod \"router-gateway-1-openshift-default-858fc4dc48-xg2z6\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") " pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.180411 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.180329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:42.318428 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.318401    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"]
Jul 27 17:15:42.320691 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:15:42.320663    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd376557_6c98_4a3c_a377_319a90face63.slice/crio-e59d584c1ef82e986b071c6369bbbea6214722120f344ab975e2c00ed5841147 WatchSource:0}: Error finding container e59d584c1ef82e986b071c6369bbbea6214722120f344ab975e2c00ed5841147: Status 404 returned error can't find the container with id e59d584c1ef82e986b071c6369bbbea6214722120f344ab975e2c00ed5841147
Jul 27 17:15:42.323211 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:42.323193    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:15:43.174763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:43.174722    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6" event={"ID":"bd376557-6c98-4a3c-a377-319a90face63","Type":"ContainerStarted","Data":"e59d584c1ef82e986b071c6369bbbea6214722120f344ab975e2c00ed5841147"}
Jul 27 17:15:45.906784 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:45.906749    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:15:45.907076 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:45.906819    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:15:45.907076 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:45.906846    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:15:46.192291 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:46.192258    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6" event={"ID":"bd376557-6c98-4a3c-a377-319a90face63","Type":"ContainerStarted","Data":"34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86"}
Jul 27 17:15:46.211879 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:46.211780    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6" podStartSLOduration=1.6285755100000001 podStartE2EDuration="5.211736431s" totalImagesPullingTime="3.583160921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:15:46.210852305 +0000 UTC m=+854.796213873" watchObservedRunningTime="2026-07-27 17:15:46.211736431 +0000 UTC m=+854.797097987"
Jul 27 17:15:47.180542 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:47.180508    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:47.185868 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:47.185828    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:47.197053 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:47.197031    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:15:47.197951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:15:47.197932    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:16:15.576163 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.576124    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"]
Jul 27 17:16:15.583643 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.583599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.587872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.587842    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-service-ca.crt\""
Jul 27 17:16:15.587872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.587868    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"kube-root-ca.crt\""
Jul 27 17:16:15.588107 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.587866    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"openshift-gw-ca-root-cert\""
Jul 27 17:16:15.588107 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.587960    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"router-gateway-1-openshift-default-dockercfg-nfq2v\""
Jul 27 17:16:15.593383 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.593352    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"]
Jul 27 17:16:15.683616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-istio-token\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683557    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-envoy\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683608    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-certs\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683637    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76365ce9-4032-4c23-b49d-109a79db8211-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683680    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76365ce9-4032-4c23-b49d-109a79db8211-istio-podinfo\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683734    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-credential-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-data\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.683887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.683855    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27qd9\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-kube-api-access-27qd9\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785182    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-istio-token\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785228    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-envoy\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785259    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-certs\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76365ce9-4032-4c23-b49d-109a79db8211-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76365ce9-4032-4c23-b49d-109a79db8211-istio-podinfo\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785365    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-credential-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785422    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-data\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27qd9\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-kube-api-access-27qd9\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.785913 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.785813    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-certs\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.786111 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.786036    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-credential-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.786337 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.786182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-data\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.786337 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.786190    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76365ce9-4032-4c23-b49d-109a79db8211-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.786498 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.786444    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-socket\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.787794 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.787769    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-envoy\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.788128 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.788104    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76365ce9-4032-4c23-b49d-109a79db8211-istio-podinfo\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.793118 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.793091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-istio-token\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.793246 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.793227    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27qd9\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-kube-api-access-27qd9\") pod \"router-gateway-1-openshift-default-54df766bd8-2npjw\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") " pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:15.921681 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:15.921592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:16.116070 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.116043    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"]
Jul 27 17:16:16.117946 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:16:16.117921    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76365ce9_4032_4c23_b49d_109a79db8211.slice/crio-3555ef28b94b0425dfe9991f2948a9fdf080b0cd5f95dccf5506f079ea35751d WatchSource:0}: Error finding container 3555ef28b94b0425dfe9991f2948a9fdf080b0cd5f95dccf5506f079ea35751d: Status 404 returned error can't find the container with id 3555ef28b94b0425dfe9991f2948a9fdf080b0cd5f95dccf5506f079ea35751d
Jul 27 17:16:16.119969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.119935    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:16:16.120076 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.120040    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:16:16.120138 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.120085    2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"}
Jul 27 17:16:16.323198 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.323157    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw" event={"ID":"76365ce9-4032-4c23-b49d-109a79db8211","Type":"ContainerStarted","Data":"9c03ba32c28f23b3b7e79076eafbf87b06999fd1eac28ec4763af810f0b74919"}
Jul 27 17:16:16.323345 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.323218    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw" event={"ID":"76365ce9-4032-4c23-b49d-109a79db8211","Type":"ContainerStarted","Data":"3555ef28b94b0425dfe9991f2948a9fdf080b0cd5f95dccf5506f079ea35751d"}
Jul 27 17:16:16.345453 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.345365    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw" podStartSLOduration=1.345342717 podStartE2EDuration="1.345342717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:16:16.343092799 +0000 UTC m=+884.928454355" watchObservedRunningTime="2026-07-27 17:16:16.345342717 +0000 UTC m=+884.930704274"
Jul 27 17:16:16.921940 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:16.921891    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:17.927427 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:17.927397    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:18.331399 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:18.331369    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:18.332363 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:18.332346    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:16:20.301558 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:20.301523    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"]
Jul 27 17:16:20.302041 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:20.301774    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6" podUID="bd376557-6c98-4a3c-a377-319a90face63" containerName="istio-proxy" containerID="cri-o://34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86" gracePeriod=30
Jul 27 17:16:22.550116 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.550092    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:16:22.647544 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647516    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-socket\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647558    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd376557-6c98-4a3c-a377-319a90face63-istio-podinfo\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-data\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647602    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-credential-socket\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-certs\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647642    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd376557-6c98-4a3c-a377-319a90face63-istiod-ca-cert\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647660    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-envoy\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-istio-token\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647755    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89lvc\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-kube-api-access-89lvc\") pod \"bd376557-6c98-4a3c-a377-319a90face63\" (UID: \"bd376557-6c98-4a3c-a377-319a90face63\") "
Jul 27 17:16:22.647970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647749    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:22.647970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647908    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:22.647970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.647922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-data" (OuterVolumeSpecName: "istio-data") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:22.648253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.648096    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-socket\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.648253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.648117    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-data\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.648253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.648132    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-credential-socket\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.648253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.648197    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:22.648253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.648228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd376557-6c98-4a3c-a377-319a90face63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:16:22.649927 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.649909    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd376557-6c98-4a3c-a377-319a90face63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:16:22.650058 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.650044    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-kube-api-access-89lvc" (OuterVolumeSpecName: "kube-api-access-89lvc") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "kube-api-access-89lvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:16:22.650381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.650355    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:16:22.650381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.650352    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-istio-token" (OuterVolumeSpecName: "istio-token") pod "bd376557-6c98-4a3c-a377-319a90face63" (UID: "bd376557-6c98-4a3c-a377-319a90face63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:16:22.749394 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.749358    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-istio-token\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.749394 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.749387    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89lvc\" (UniqueName: \"kubernetes.io/projected/bd376557-6c98-4a3c-a377-319a90face63-kube-api-access-89lvc\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.749394 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.749401    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd376557-6c98-4a3c-a377-319a90face63-istio-podinfo\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.749656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.749412    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-workload-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.749656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.749424    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd376557-6c98-4a3c-a377-319a90face63-istiod-ca-cert\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:22.749656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:22.749435    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd376557-6c98-4a3c-a377-319a90face63-istio-envoy\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:16:23.355108 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.355073    2579 generic.go:356] "Generic (PLEG): container finished" podID="bd376557-6c98-4a3c-a377-319a90face63" containerID="34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86" exitCode=0
Jul 27 17:16:23.355277 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.355118    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"
Jul 27 17:16:23.355277 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.355134    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6" event={"ID":"bd376557-6c98-4a3c-a377-319a90face63","Type":"ContainerDied","Data":"34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86"}
Jul 27 17:16:23.355277 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.355177    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6" event={"ID":"bd376557-6c98-4a3c-a377-319a90face63","Type":"ContainerDied","Data":"e59d584c1ef82e986b071c6369bbbea6214722120f344ab975e2c00ed5841147"}
Jul 27 17:16:23.355277 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.355198    2579 scope.go:117] "RemoveContainer" containerID="34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86"
Jul 27 17:16:23.364699 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.364680    2579 scope.go:117] "RemoveContainer" containerID="34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86"
Jul 27 17:16:23.364973 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:16:23.364954    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86\": container with ID starting with 34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86 not found: ID does not exist" containerID="34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86"
Jul 27 17:16:23.365019 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.364982    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86"} err="failed to get container status \"34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86\": rpc error: code = NotFound desc = could not find container \"34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86\": container with ID starting with 34145649cfc9b0f0111d57192f7127b26e1b560efceca499221bed70ad613a86 not found: ID does not exist"
Jul 27 17:16:23.376358 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.376332    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"]
Jul 27 17:16:23.382395 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.382376    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-49b5edd9/router-gateway-1-openshift-default-858fc4dc48-xg2z6"]
Jul 27 17:16:23.892950 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:23.892908    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd376557-6c98-4a3c-a377-319a90face63" path="/var/lib/kubelet/pods/bd376557-6c98-4a3c-a377-319a90face63/volumes"
Jul 27 17:16:31.785921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:31.785895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:16:31.786626 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:31.786606    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:16:39.795429 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.795395    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:16:39.795855 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.795729    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd376557-6c98-4a3c-a377-319a90face63" containerName="istio-proxy"
Jul 27 17:16:39.795855 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.795740    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd376557-6c98-4a3c-a377-319a90face63" containerName="istio-proxy"
Jul 27 17:16:39.795855 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.795790    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd376557-6c98-4a3c-a377-319a90face63" containerName="istio-proxy"
Jul 27 17:16:39.799945 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.799926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.802577 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.802503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 27 17:16:39.802718 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.802622    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Jul 27 17:16:39.802718 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.802628    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"seaweedfs-s3-creds\""
Jul 27 17:16:39.802718 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.802669    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Jul 27 17:16:39.803392 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.803377    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-tzkd5\""
Jul 27 17:16:39.803457 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.803440    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-cq8pb\""
Jul 27 17:16:39.810851 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.810829    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:16:39.898403 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfdk6\" (UniqueName: \"kubernetes.io/projected/fc2b5d07-1c57-424a-a149-dc45662f85a8-kube-api-access-qfdk6\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.898403 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898406    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.898629 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.898629 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898458    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.898629 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2b5d07-1c57-424a-a149-dc45662f85a8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.898629 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898503    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.898629 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.898521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999214 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999178    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfdk6\" (UniqueName: \"kubernetes.io/projected/fc2b5d07-1c57-424a-a149-dc45662f85a8-kube-api-access-qfdk6\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999402 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999467 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999399    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999467 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999435    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999467 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2b5d07-1c57-424a-a149-dc45662f85a8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999524    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999556    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999560    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999671    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999861 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999812    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:39.999915 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:39.999869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:40.001758 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:40.001741    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:40.001945 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:40.001927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2b5d07-1c57-424a-a149-dc45662f85a8-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:40.007212 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:40.007195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfdk6\" (UniqueName: \"kubernetes.io/projected/fc2b5d07-1c57-424a-a149-dc45662f85a8-kube-api-access-qfdk6\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:40.111606 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:40.111503    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:16:40.270259 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:40.270235    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:16:40.272606 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:16:40.272581    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2b5d07_1c57_424a_a149_dc45662f85a8.slice/crio-a27fd6bf18dec57c7ce77696ad9adb0bf055c55415f40a978d1ed3a28e10401d WatchSource:0}: Error finding container a27fd6bf18dec57c7ce77696ad9adb0bf055c55415f40a978d1ed3a28e10401d: Status 404 returned error can't find the container with id a27fd6bf18dec57c7ce77696ad9adb0bf055c55415f40a978d1ed3a28e10401d
Jul 27 17:16:40.431133 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:40.431045    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fc2b5d07-1c57-424a-a149-dc45662f85a8","Type":"ContainerStarted","Data":"a27fd6bf18dec57c7ce77696ad9adb0bf055c55415f40a978d1ed3a28e10401d"}
Jul 27 17:16:45.458112 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:16:45.458081    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fc2b5d07-1c57-424a-a149-dc45662f85a8","Type":"ContainerStarted","Data":"33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6"}
Jul 27 17:17:00.525976 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:00.525943    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerID="33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6" exitCode=0
Jul 27 17:17:00.526411 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:00.526019    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fc2b5d07-1c57-424a-a149-dc45662f85a8","Type":"ContainerDied","Data":"33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6"}
Jul 27 17:17:02.981950 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:02.981250    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"]
Jul 27 17:17:02.985083 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:02.985060    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:02.989212 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:02.988772    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"default-dockercfg-vfvxf\""
Jul 27 17:17:02.989212 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:02.989042    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-gateway-section-name-propagation-94799d44\"/\"gw-sec2774c263d49959f50d9eebc552e13bf9-kserve-self-signed-certs\""
Jul 27 17:17:02.994625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:02.994561    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"]
Jul 27 17:17:03.078332 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.078281    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.078529 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.078399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.078529 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.078508    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c078f87-694e-45d0-899f-ec8c0d1ad551-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.078661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.078559    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.078661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.078593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.078661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.078622    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkjvt\" (UniqueName: \"kubernetes.io/projected/9c078f87-694e-45d0-899f-ec8c0d1ad551-kube-api-access-zkjvt\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179291 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179254    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c078f87-694e-45d0-899f-ec8c0d1ad551-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179360    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkjvt\" (UniqueName: \"kubernetes.io/projected/9c078f87-694e-45d0-899f-ec8c0d1ad551-kube-api-access-zkjvt\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179499    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179866 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-tmp-dir\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.179938 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.179851    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-model-cache\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.180369 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.180075    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-home\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.183201 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.183175    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c078f87-694e-45d0-899f-ec8c0d1ad551-tls-certs\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.184227 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.184179    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-dshm\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.189877 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.189826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkjvt\" (UniqueName: \"kubernetes.io/projected/9c078f87-694e-45d0-899f-ec8c0d1ad551-kube-api-access-zkjvt\") pod \"gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") " pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.301900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.301797    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:03.466558 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.466506    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"]
Jul 27 17:17:03.469676 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:17:03.469641    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c078f87_694e_45d0_899f_ec8c0d1ad551.slice/crio-0e3d71204aa05263809163070900f5685cbf7fc4c14b018bbfa935f41e4796a1 WatchSource:0}: Error finding container 0e3d71204aa05263809163070900f5685cbf7fc4c14b018bbfa935f41e4796a1: Status 404 returned error can't find the container with id 0e3d71204aa05263809163070900f5685cbf7fc4c14b018bbfa935f41e4796a1
Jul 27 17:17:03.549544 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:03.549446    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" event={"ID":"9c078f87-694e-45d0-899f-ec8c0d1ad551","Type":"ContainerStarted","Data":"0e3d71204aa05263809163070900f5685cbf7fc4c14b018bbfa935f41e4796a1"}
Jul 27 17:17:06.586415 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:06.586306    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" event={"ID":"9c078f87-694e-45d0-899f-ec8c0d1ad551","Type":"ContainerStarted","Data":"6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81"}
Jul 27 17:17:06.606226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:06.606153    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" podStartSLOduration=1.6683152639999999 podStartE2EDuration="4.606134483s" totalImagesPullingTime="2.937819219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:17:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:17:06.605639874 +0000 UTC m=+935.191001431" watchObservedRunningTime="2026-07-27 17:17:06.606134483 +0000 UTC m=+935.191496042"
Jul 27 17:17:07.302188 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:07.302154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:11.786039 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:11.785996    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"]
Jul 27 17:17:11.792706 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:11.792553    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"]
Jul 27 17:17:11.792887 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:11.792799    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw" podUID="76365ce9-4032-4c23-b49d-109a79db8211" containerName="istio-proxy" containerID="cri-o://9c03ba32c28f23b3b7e79076eafbf87b06999fd1eac28ec4763af810f0b74919" gracePeriod=30
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:17:11.799873    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81" cmd=["/bin/sleep","15"]
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:11.799943    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" message="" err=<
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:17:11.799959    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" podUID="9c078f87-694e-45d0-899f-ec8c0d1ad551" containerName="main" containerID="cri-o://6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81"
Jul 27 17:17:11.800160 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:11.800007    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" podUID="9c078f87-694e-45d0-899f-ec8c0d1ad551" containerName="main" containerID="cri-o://6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81" gracePeriod=60
Jul 27 17:17:14.636189 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:14.636152    2579 generic.go:356] "Generic (PLEG): container finished" podID="9c078f87-694e-45d0-899f-ec8c0d1ad551" containerID="6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81" exitCode=0
Jul 27 17:17:14.636609 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:14.636209    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" event={"ID":"9c078f87-694e-45d0-899f-ec8c0d1ad551","Type":"ContainerDied","Data":"6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81"}
Jul 27 17:17:15.430454 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.402423    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:15.523498 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523442    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkjvt\" (UniqueName: \"kubernetes.io/projected/9c078f87-694e-45d0-899f-ec8c0d1ad551-kube-api-access-zkjvt\") pod \"9c078f87-694e-45d0-899f-ec8c0d1ad551\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") "
Jul 27 17:17:15.523498 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523491    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-home\") pod \"9c078f87-694e-45d0-899f-ec8c0d1ad551\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") "
Jul 27 17:17:15.523706 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523564    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-dshm\") pod \"9c078f87-694e-45d0-899f-ec8c0d1ad551\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") "
Jul 27 17:17:15.523706 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523593    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c078f87-694e-45d0-899f-ec8c0d1ad551-tls-certs\") pod \"9c078f87-694e-45d0-899f-ec8c0d1ad551\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") "
Jul 27 17:17:15.523706 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523627    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-model-cache\") pod \"9c078f87-694e-45d0-899f-ec8c0d1ad551\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") "
Jul 27 17:17:15.523706 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523656    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-tmp-dir\") pod \"9c078f87-694e-45d0-899f-ec8c0d1ad551\" (UID: \"9c078f87-694e-45d0-899f-ec8c0d1ad551\") "
Jul 27 17:17:15.523979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523832    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-home" (OuterVolumeSpecName: "home") pod "9c078f87-694e-45d0-899f-ec8c0d1ad551" (UID: "9c078f87-694e-45d0-899f-ec8c0d1ad551"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.523979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.523923    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.524126 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.524024    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c078f87-694e-45d0-899f-ec8c0d1ad551" (UID: "9c078f87-694e-45d0-899f-ec8c0d1ad551"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.524354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.524172    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9c078f87-694e-45d0-899f-ec8c0d1ad551" (UID: "9c078f87-694e-45d0-899f-ec8c0d1ad551"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.526145 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.526116    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c078f87-694e-45d0-899f-ec8c0d1ad551-kube-api-access-zkjvt" (OuterVolumeSpecName: "kube-api-access-zkjvt") pod "9c078f87-694e-45d0-899f-ec8c0d1ad551" (UID: "9c078f87-694e-45d0-899f-ec8c0d1ad551"). InnerVolumeSpecName "kube-api-access-zkjvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:17:15.526263 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.526237    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c078f87-694e-45d0-899f-ec8c0d1ad551-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9c078f87-694e-45d0-899f-ec8c0d1ad551" (UID: "9c078f87-694e-45d0-899f-ec8c0d1ad551"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:17:15.526643 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.526621    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-dshm" (OuterVolumeSpecName: "dshm") pod "9c078f87-694e-45d0-899f-ec8c0d1ad551" (UID: "9c078f87-694e-45d0-899f-ec8c0d1ad551"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.625121 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.625088    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.625121 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.625125    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.625305 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.625140    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkjvt\" (UniqueName: \"kubernetes.io/projected/9c078f87-694e-45d0-899f-ec8c0d1ad551-kube-api-access-zkjvt\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.625305 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.625156    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c078f87-694e-45d0-899f-ec8c0d1ad551-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.625305 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.625171    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c078f87-694e-45d0-899f-ec8c0d1ad551-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.644691 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.644656    2579 generic.go:356] "Generic (PLEG): container finished" podID="76365ce9-4032-4c23-b49d-109a79db8211" containerID="9c03ba32c28f23b3b7e79076eafbf87b06999fd1eac28ec4763af810f0b74919" exitCode=0
Jul 27 17:17:15.645106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.644741    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw" event={"ID":"76365ce9-4032-4c23-b49d-109a79db8211","Type":"ContainerDied","Data":"9c03ba32c28f23b3b7e79076eafbf87b06999fd1eac28ec4763af810f0b74919"}
Jul 27 17:17:15.646250 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.646205    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7" event={"ID":"9c078f87-694e-45d0-899f-ec8c0d1ad551","Type":"ContainerDied","Data":"0e3d71204aa05263809163070900f5685cbf7fc4c14b018bbfa935f41e4796a1"}
Jul 27 17:17:15.646250 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.646233    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"
Jul 27 17:17:15.646510 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.646262    2579 scope.go:117] "RemoveContainer" containerID="6ad2e0518bcdcec0cb7627d7f318e56231b7fcd587bfa6874a6856b42a693a81"
Jul 27 17:17:15.671681 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.671647    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"]
Jul 27 17:17:15.675806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.675783    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/gw-section-name-router-with-gat-f1d92d0f-kserve-5f5b776bbf928p7"]
Jul 27 17:17:15.713212 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.713191    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:17:15.827544 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827501    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76365ce9-4032-4c23-b49d-109a79db8211-istio-podinfo\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.827712 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827571    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76365ce9-4032-4c23-b49d-109a79db8211-istiod-ca-cert\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.827712 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827641    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-istio-token\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.827712 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827674    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-data\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.827712 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827707    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27qd9\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-kube-api-access-27qd9\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.828026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827843    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-certs\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.828026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827873    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-credential-socket\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.828026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827933    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-envoy\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.828026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.827994    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-socket\") pod \"76365ce9-4032-4c23-b49d-109a79db8211\" (UID: \"76365ce9-4032-4c23-b49d-109a79db8211\") "
Jul 27 17:17:15.828026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.828005    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76365ce9-4032-4c23-b49d-109a79db8211-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 27 17:17:15.828348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.828236    2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76365ce9-4032-4c23-b49d-109a79db8211-istiod-ca-cert\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.828450 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.828430    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.828539 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.828523    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.828645 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.828597    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-data" (OuterVolumeSpecName: "istio-data") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.828645 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.828660    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.830373 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.830228    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76365ce9-4032-4c23-b49d-109a79db8211-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 27 17:17:15.830531 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.830467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-kube-api-access-27qd9" (OuterVolumeSpecName: "kube-api-access-27qd9") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "kube-api-access-27qd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:17:15.830733 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.830701    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-istio-token" (OuterVolumeSpecName: "istio-token") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:17:15.831119 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.831094    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "76365ce9-4032-4c23-b49d-109a79db8211" (UID: "76365ce9-4032-4c23-b49d-109a79db8211"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:17:15.895294 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.895257    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c078f87-694e-45d0-899f-ec8c0d1ad551" path="/var/lib/kubelet/pods/9c078f87-694e-45d0-899f-ec8c0d1ad551/volumes"
Jul 27 17:17:15.929616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929586    2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-istio-token\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929619    2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-data\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929635    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27qd9\" (UniqueName: \"kubernetes.io/projected/76365ce9-4032-4c23-b49d-109a79db8211-kube-api-access-27qd9\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929650    2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929666    2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-credential-socket\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929684    2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-istio-envoy\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929695    2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76365ce9-4032-4c23-b49d-109a79db8211-workload-socket\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:15.929804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:15.929704    2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76365ce9-4032-4c23-b49d-109a79db8211-istio-podinfo\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:17:16.654398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:16.654318    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"
Jul 27 17:17:16.654398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:16.654332    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw" event={"ID":"76365ce9-4032-4c23-b49d-109a79db8211","Type":"ContainerDied","Data":"3555ef28b94b0425dfe9991f2948a9fdf080b0cd5f95dccf5506f079ea35751d"}
Jul 27 17:17:16.654938 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:16.654415    2579 scope.go:117] "RemoveContainer" containerID="9c03ba32c28f23b3b7e79076eafbf87b06999fd1eac28ec4763af810f0b74919"
Jul 27 17:17:16.673322 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:16.673293    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"]
Jul 27 17:17:16.677839 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:16.677813    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-gateway-section-name-propagation-94799d44/router-gateway-1-openshift-default-54df766bd8-2npjw"]
Jul 27 17:17:17.893205 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:17.893174    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76365ce9-4032-4c23-b49d-109a79db8211" path="/var/lib/kubelet/pods/76365ce9-4032-4c23-b49d-109a79db8211/volumes"
Jul 27 17:17:29.908311 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.908275    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"]
Jul 27 17:17:29.908999 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.908858    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76365ce9-4032-4c23-b49d-109a79db8211" containerName="istio-proxy"
Jul 27 17:17:29.908999 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.908883    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="76365ce9-4032-4c23-b49d-109a79db8211" containerName="istio-proxy"
Jul 27 17:17:29.908999 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.908904    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c078f87-694e-45d0-899f-ec8c0d1ad551" containerName="main"
Jul 27 17:17:29.908999 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.908914    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c078f87-694e-45d0-899f-ec8c0d1ad551" containerName="main"
Jul 27 17:17:29.909290 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.909012    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="76365ce9-4032-4c23-b49d-109a79db8211" containerName="istio-proxy"
Jul 27 17:17:29.909290 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.909027    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c078f87-694e-45d0-899f-ec8c0d1ad551" containerName="main"
Jul 27 17:17:29.951732 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.951697    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"]
Jul 27 17:17:29.951895 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.951773    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:29.954827 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.954789    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 27 17:17:29.955027 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.954880    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-d7h8x\""
Jul 27 17:17:29.955176 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.955158    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Jul 27 17:17:29.955259 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.955236    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Jul 27 17:17:29.955332 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:29.955290    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 27 17:17:30.053220 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tmp-dir\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.053402 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mshnl\" (UniqueName: \"kubernetes.io/projected/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kube-api-access-mshnl\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.053402 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053269    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tls-certs\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.053402 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053312    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-home\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.053402 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053342    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kserve-provision-location\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.053608 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053399    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-dshm\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.053608 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.053440    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-model-cache\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.154769 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154731    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-home\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.154936 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kserve-provision-location\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.154936 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154834    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-dshm\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.154936 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154859    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-model-cache\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.154936 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154925    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tmp-dir\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.155170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154950    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshnl\" (UniqueName: \"kubernetes.io/projected/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kube-api-access-mshnl\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.155170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.154978    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tls-certs\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.155283 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.155174    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-home\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.155283 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.155193    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kserve-provision-location\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.155758 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.155715    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-model-cache\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.155758 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.155751    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tmp-dir\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.158432 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.158379    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-dshm\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.158573 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.158536    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tls-certs\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.175045 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.175021    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshnl\" (UniqueName: \"kubernetes.io/projected/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kube-api-access-mshnl\") pod \"auth-enabled-test-kserve-7bf965476d-xnzzc\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:30.263708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:30.263676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:32.411226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:32.411200    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"]
Jul 27 17:17:32.563936 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:17:32.563856    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f334cb5_5e11_45ed_a080_7d1d7e6bd4e5.slice/crio-2eaf59f34db32fe600d3be0330c419d8397f17c86445545bd247d2b10f2b3072 WatchSource:0}: Error finding container 2eaf59f34db32fe600d3be0330c419d8397f17c86445545bd247d2b10f2b3072: Status 404 returned error can't find the container with id 2eaf59f34db32fe600d3be0330c419d8397f17c86445545bd247d2b10f2b3072
Jul 27 17:17:32.585220 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:32.585198    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"seaweedfs-s3-creds\""
Jul 27 17:17:32.739423 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:32.739386    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" event={"ID":"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5","Type":"ContainerStarted","Data":"c0e9dcdd979467efb77b8a6c1f386239945cc189abb84fa0feb7596512731c92"}
Jul 27 17:17:32.739587 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:32.739441    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" event={"ID":"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5","Type":"ContainerStarted","Data":"2eaf59f34db32fe600d3be0330c419d8397f17c86445545bd247d2b10f2b3072"}
Jul 27 17:17:33.748902 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:33.748730    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fc2b5d07-1c57-424a-a149-dc45662f85a8","Type":"ContainerStarted","Data":"316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7"}
Jul 27 17:17:33.774020 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:33.773960    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podStartSLOduration=2.022572625 podStartE2EDuration="54.773944189s" totalImagesPullingTime="37.238471944s" totalInitContainerRuntime="15.51289962s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:17:33.772304769 +0000 UTC m=+962.357666325" watchObservedRunningTime="2026-07-27 17:17:33.773944189 +0000 UTC m=+962.359305753"
Jul 27 17:17:40.787651 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:40.787616    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerID="c0e9dcdd979467efb77b8a6c1f386239945cc189abb84fa0feb7596512731c92" exitCode=0
Jul 27 17:17:40.788092 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:40.787659    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" event={"ID":"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5","Type":"ContainerDied","Data":"c0e9dcdd979467efb77b8a6c1f386239945cc189abb84fa0feb7596512731c92"}
Jul 27 17:17:41.794388 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:41.794347    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" event={"ID":"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5","Type":"ContainerStarted","Data":"0e0db2be0d595afd123fb4680f33b5df8feffa3273303b491e39ebf0ffe1cbc2"}
Jul 27 17:17:41.819207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:41.819148    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podStartSLOduration=5.025382511 podStartE2EDuration="12.819130545s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.793748034s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:17:41.816698825 +0000 UTC m=+970.402060392" watchObservedRunningTime="2026-07-27 17:17:41.819130545 +0000 UTC m=+970.404492100"
Jul 27 17:17:42.264568 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:42.264519    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:50.264285 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:50.264241    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:17:50.265987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:17:50.265949    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:18:00.264846 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:18:00.264748    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:18:10.264410 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:18:10.264358    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:18:20.264251 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:18:20.264206    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:18:30.264563 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:18:30.264503    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:18:40.265170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:18:40.265121    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:18:50.264787 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:18:50.264729    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:19:00.264484 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:00.264437    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:19:10.264431 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:10.264377    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:19:20.264387 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:20.264344    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:19:30.264916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:30.264822    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:19:40.264049 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:40.264008    2579 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:19:50.274404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:50.274371    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:19:50.282100 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:50.282074    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:19:54.941723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:19:54.941467    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"]
Jul 27 17:20:09.964207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:09.964148    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" containerID="cri-o://0e0db2be0d595afd123fb4680f33b5df8feffa3273303b491e39ebf0ffe1cbc2" gracePeriod=45
Jul 27 17:20:10.275987 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.275896    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": dial tcp 10.134.0.30:8000: connect: connection refused"
Jul 27 17:20:10.512000 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.511967    2579 generic.go:356] "Generic (PLEG): container finished" podID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerID="0e0db2be0d595afd123fb4680f33b5df8feffa3273303b491e39ebf0ffe1cbc2" exitCode=0
Jul 27 17:20:10.512133 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.512036    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" event={"ID":"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5","Type":"ContainerDied","Data":"0e0db2be0d595afd123fb4680f33b5df8feffa3273303b491e39ebf0ffe1cbc2"}
Jul 27 17:20:10.629732 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.629706    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:20:10.737750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kserve-provision-location\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.737750 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737751    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-model-cache\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.737979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mshnl\" (UniqueName: \"kubernetes.io/projected/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kube-api-access-mshnl\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.737979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737823    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-dshm\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.737979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tls-certs\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.737979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737901    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tmp-dir\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.737979 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.737943    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-home\") pod \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\" (UID: \"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5\") "
Jul 27 17:20:10.738200 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.738041    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:10.738267 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.738249    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:10.738669 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.738634    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-home" (OuterVolumeSpecName: "home") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:10.740290 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.740265    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-dshm" (OuterVolumeSpecName: "dshm") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:10.740514 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.740491    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kube-api-access-mshnl" (OuterVolumeSpecName: "kube-api-access-mshnl") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "kube-api-access-mshnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:20:10.740514 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.740501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:20:10.756258 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.756220    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:10.805450 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.805377    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" (UID: "7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:10.839013 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.838971    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kserve-provision-location\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:10.839013 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.839010    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mshnl\" (UniqueName: \"kubernetes.io/projected/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-kube-api-access-mshnl\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:10.839181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.839025    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:10.839181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.839039    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:10.839181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.839052    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:10.839181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:10.839062    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:11.518657 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.518632    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"
Jul 27 17:20:11.519194 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.518623    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc" event={"ID":"7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5","Type":"ContainerDied","Data":"2eaf59f34db32fe600d3be0330c419d8397f17c86445545bd247d2b10f2b3072"}
Jul 27 17:20:11.519194 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.518774    2579 scope.go:117] "RemoveContainer" containerID="0e0db2be0d595afd123fb4680f33b5df8feffa3273303b491e39ebf0ffe1cbc2"
Jul 27 17:20:11.531967 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.531943    2579 scope.go:117] "RemoveContainer" containerID="c0e9dcdd979467efb77b8a6c1f386239945cc189abb84fa0feb7596512731c92"
Jul 27 17:20:11.544736 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.544714    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"]
Jul 27 17:20:11.548702 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.548683    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-7bf965476d-xnzzc"]
Jul 27 17:20:11.892315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:11.892234    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" path="/var/lib/kubelet/pods/7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5/volumes"
Jul 27 17:20:18.477413 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:18.477373    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:20:33.499730 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:33.499665    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerName="main" containerID="cri-o://316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7" gracePeriod=45
Jul 27 17:20:34.150958 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.150935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:20:34.224312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224221    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2b5d07-1c57-424a-a149-dc45662f85a8-tls-certs\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224279    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-model-cache\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224304    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-kserve-provision-location\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224328    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-home\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224352    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-tmp-dir\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224367    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-dshm\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224393    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfdk6\" (UniqueName: \"kubernetes.io/projected/fc2b5d07-1c57-424a-a149-dc45662f85a8-kube-api-access-qfdk6\") pod \"fc2b5d07-1c57-424a-a149-dc45662f85a8\" (UID: \"fc2b5d07-1c57-424a-a149-dc45662f85a8\") "
Jul 27 17:20:34.224771 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224623    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:34.224938 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.224906    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-home" (OuterVolumeSpecName: "home") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:34.226695 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.226669    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2b5d07-1c57-424a-a149-dc45662f85a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:20:34.226780 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.226727    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-dshm" (OuterVolumeSpecName: "dshm") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:34.226904 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.226883    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2b5d07-1c57-424a-a149-dc45662f85a8-kube-api-access-qfdk6" (OuterVolumeSpecName: "kube-api-access-qfdk6") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "kube-api-access-qfdk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:20:34.241252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.241218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:34.285385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.285344    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc2b5d07-1c57-424a-a149-dc45662f85a8" (UID: "fc2b5d07-1c57-424a-a149-dc45662f85a8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:20:34.325251 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325227    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-kserve-provision-location\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.325251 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325251    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.325378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325263    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.325378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325272    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.325378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325280    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfdk6\" (UniqueName: \"kubernetes.io/projected/fc2b5d07-1c57-424a-a149-dc45662f85a8-kube-api-access-qfdk6\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.325378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325289    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc2b5d07-1c57-424a-a149-dc45662f85a8-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.325378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.325297    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc2b5d07-1c57-424a-a149-dc45662f85a8-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:20:34.618349 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.618262    2579 generic.go:356] "Generic (PLEG): container finished" podID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerID="316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7" exitCode=0
Jul 27 17:20:34.618349 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.618308    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"
Jul 27 17:20:34.618760 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.618308    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fc2b5d07-1c57-424a-a149-dc45662f85a8","Type":"ContainerDied","Data":"316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7"}
Jul 27 17:20:34.618760 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.618445    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1" event={"ID":"fc2b5d07-1c57-424a-a149-dc45662f85a8","Type":"ContainerDied","Data":"a27fd6bf18dec57c7ce77696ad9adb0bf055c55415f40a978d1ed3a28e10401d"}
Jul 27 17:20:34.618760 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.618497    2579 scope.go:117] "RemoveContainer" containerID="316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7"
Jul 27 17:20:34.626976 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.626957    2579 scope.go:117] "RemoveContainer" containerID="33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6"
Jul 27 17:20:34.634505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.634460    2579 scope.go:117] "RemoveContainer" containerID="316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7"
Jul 27 17:20:34.634764 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:20:34.634745    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7\": container with ID starting with 316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7 not found: ID does not exist" containerID="316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7"
Jul 27 17:20:34.634830 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.634772    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7"} err="failed to get container status \"316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7\": rpc error: code = NotFound desc = could not find container \"316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7\": container with ID starting with 316fae8e5d96a1649a7504818dd40e3bce9b191b9e11d6a0dfda184f263363e7 not found: ID does not exist"
Jul 27 17:20:34.634830 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.634789    2579 scope.go:117] "RemoveContainer" containerID="33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6"
Jul 27 17:20:34.635025 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:20:34.635007    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6\": container with ID starting with 33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6 not found: ID does not exist" containerID="33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6"
Jul 27 17:20:34.635070 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.635037    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6"} err="failed to get container status \"33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6\": rpc error: code = NotFound desc = could not find container \"33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6\": container with ID starting with 33e1d0d8ca25fc94f45cb6ae94e4b3ada74e948ffe019e00764fe978479b89d6 not found: ID does not exist"
Jul 27 17:20:34.640867 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.640846    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:20:34.647316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:34.647297    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0-1"]
Jul 27 17:20:35.892662 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:35.892628    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" path="/var/lib/kubelet/pods/fc2b5d07-1c57-424a-a149-dc45662f85a8/volumes"
Jul 27 17:20:50.391335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391296    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"]
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391627    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391638    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391654    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerName="storage-initializer"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391659    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerName="storage-initializer"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391670    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerName="main"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391675    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerName="main"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391685    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="storage-initializer"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391690    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="storage-initializer"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391738    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc2b5d07-1c57-424a-a149-dc45662f85a8" containerName="main"
Jul 27 17:20:50.391805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.391747    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f334cb5-5e11-45ed-a080-7d1d7e6bd4e5" containerName="main"
Jul 27 17:20:50.396088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.396069    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.400635 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.400612    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Jul 27 17:20:50.400761 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.400610    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Jul 27 17:20:50.400761 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.400618    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Jul 27 17:20:50.400761 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.400671    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-vlnf6\""
Jul 27 17:20:50.403520 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.403498    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"]
Jul 27 17:20:50.452192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.452164    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.452192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.452195    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeda0c23-2dad-476c-9df4-47afadef0068-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.452381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.452214    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.452381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.452230    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.452381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.452246    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctqw\" (UniqueName: \"kubernetes.io/projected/aeda0c23-2dad-476c-9df4-47afadef0068-kube-api-access-sctqw\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.452381 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.452307    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.553719 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.553684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.553900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.553736    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.553900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.553767    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeda0c23-2dad-476c-9df4-47afadef0068-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.553900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.553786    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.553900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.553802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.553900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.553824    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctqw\" (UniqueName: \"kubernetes.io/projected/aeda0c23-2dad-476c-9df4-47afadef0068-kube-api-access-sctqw\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.554252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.554222    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.554366 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.554271    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.554366 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.554307    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.556140 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.556121    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.556398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.556381    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeda0c23-2dad-476c-9df4-47afadef0068-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.561241 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.561221    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctqw\" (UniqueName: \"kubernetes.io/projected/aeda0c23-2dad-476c-9df4-47afadef0068-kube-api-access-sctqw\") pod \"scheduler-inline-config-test-kserve-8655bc757-kldr7\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.655708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.655632    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"]
Jul 27 17:20:50.659160 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.659141    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.661501 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.661467    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-xfnzz\""
Jul 27 17:20:50.667832 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.667810    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"]
Jul 27 17:20:50.706600 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.706573    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:50.755607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.755571    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlj2\" (UniqueName: \"kubernetes.io/projected/77e39f35-91b0-4808-a465-0c881b026490-kube-api-access-jdlj2\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557749p4\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.755816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.755623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e39f35-91b0-4808-a465-0c881b026490-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557749p4\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.835601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.835578    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"]
Jul 27 17:20:50.837645 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:20:50.837611    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeda0c23_2dad_476c_9df4_47afadef0068.slice/crio-e6579c56f7976f660eaff6e9aa99d656a586f632079d5bf5600541626059b1af WatchSource:0}: Error finding container e6579c56f7976f660eaff6e9aa99d656a586f632079d5bf5600541626059b1af: Status 404 returned error can't find the container with id e6579c56f7976f660eaff6e9aa99d656a586f632079d5bf5600541626059b1af
Jul 27 17:20:50.856450 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.856427    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlj2\" (UniqueName: \"kubernetes.io/projected/77e39f35-91b0-4808-a465-0c881b026490-kube-api-access-jdlj2\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557749p4\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.856573 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.856498    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e39f35-91b0-4808-a465-0c881b026490-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557749p4\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.858956 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.858935    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e39f35-91b0-4808-a465-0c881b026490-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557749p4\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.864591 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.864568    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlj2\" (UniqueName: \"kubernetes.io/projected/77e39f35-91b0-4808-a465-0c881b026490-kube-api-access-jdlj2\") pod \"scheduler-inline-config-test-kserve-router-scheduler-7d557749p4\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:50.970561 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:50.970530    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:51.101577 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.101491    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"]
Jul 27 17:20:51.103661 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:20:51.103635    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e39f35_91b0_4808_a465_0c881b026490.slice/crio-a02c0771e5d9934289e926db7ed06141e86540f9464a17953299c6cf14eb2fe5 WatchSource:0}: Error finding container a02c0771e5d9934289e926db7ed06141e86540f9464a17953299c6cf14eb2fe5: Status 404 returned error can't find the container with id a02c0771e5d9934289e926db7ed06141e86540f9464a17953299c6cf14eb2fe5
Jul 27 17:20:51.105566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.105552    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:20:51.697391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.697354    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" event={"ID":"77e39f35-91b0-4808-a465-0c881b026490","Type":"ContainerStarted","Data":"a02c0771e5d9934289e926db7ed06141e86540f9464a17953299c6cf14eb2fe5"}
Jul 27 17:20:51.698670 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.698650    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" event={"ID":"aeda0c23-2dad-476c-9df4-47afadef0068","Type":"ContainerStarted","Data":"1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"}
Jul 27 17:20:51.698767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.698688    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" event={"ID":"aeda0c23-2dad-476c-9df4-47afadef0068","Type":"ContainerStarted","Data":"e6579c56f7976f660eaff6e9aa99d656a586f632079d5bf5600541626059b1af"}
Jul 27 17:20:51.707350 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.707324    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:20:51.717637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:51.717594    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" podStartSLOduration=1.717576869 podStartE2EDuration="1.717576869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:20:51.716017087 +0000 UTC m=+1160.301378644" watchObservedRunningTime="2026-07-27 17:20:51.717576869 +0000 UTC m=+1160.302938431"
Jul 27 17:20:53.708911 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:53.708877    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" event={"ID":"77e39f35-91b0-4808-a465-0c881b026490","Type":"ContainerStarted","Data":"2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b"}
Jul 27 17:20:53.709335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:53.709144    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:20:53.727767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:20:53.727726    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" podStartSLOduration=1.378894237 podStartE2EDuration="3.727712287s" totalImagesPullingTime="2.34881805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:20:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:20:53.725619893 +0000 UTC m=+1162.310981450" watchObservedRunningTime="2026-07-27 17:20:53.727712287 +0000 UTC m=+1162.313073883"
Jul 27 17:21:00.706847 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:00.706811    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:21:00.718743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:00.718721    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:21:00.751508 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:00.751465    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:21:24.714803 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:24.714772    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:21:26.026164 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.026118    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"]
Jul 27 17:21:26.027606 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.027580    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"]
Jul 27 17:21:26.035650 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:21:26.035606    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:21:26.035650 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:21:26.035650 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:21:26.035650 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428" cmd=["/bin/sleep","15"]
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.035674    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" message="" err=<
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:21:26.035691    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" podUID="aeda0c23-2dad-476c-9df4-47afadef0068" containerName="main" containerID="cri-o://1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"
Jul 27 17:21:26.035877 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.035745    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" podUID="aeda0c23-2dad-476c-9df4-47afadef0068" containerName="main" containerID="cri-o://1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428" gracePeriod=60
Jul 27 17:21:26.382307 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.382280    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:21:26.453463 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453432    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sctqw\" (UniqueName: \"kubernetes.io/projected/aeda0c23-2dad-476c-9df4-47afadef0068-kube-api-access-sctqw\") pod \"aeda0c23-2dad-476c-9df4-47afadef0068\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") "
Jul 27 17:21:26.453676 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453498    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-model-cache\") pod \"aeda0c23-2dad-476c-9df4-47afadef0068\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") "
Jul 27 17:21:26.453676 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453518    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-tmp-dir\") pod \"aeda0c23-2dad-476c-9df4-47afadef0068\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") "
Jul 27 17:21:26.453676 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453624    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-home\") pod \"aeda0c23-2dad-476c-9df4-47afadef0068\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") "
Jul 27 17:21:26.453853 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453680    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-dshm\") pod \"aeda0c23-2dad-476c-9df4-47afadef0068\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") "
Jul 27 17:21:26.453853 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453710    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeda0c23-2dad-476c-9df4-47afadef0068-tls-certs\") pod \"aeda0c23-2dad-476c-9df4-47afadef0068\" (UID: \"aeda0c23-2dad-476c-9df4-47afadef0068\") "
Jul 27 17:21:26.453853 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453781    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-model-cache" (OuterVolumeSpecName: "model-cache") pod "aeda0c23-2dad-476c-9df4-47afadef0068" (UID: "aeda0c23-2dad-476c-9df4-47afadef0068"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:21:26.453853 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453823    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aeda0c23-2dad-476c-9df4-47afadef0068" (UID: "aeda0c23-2dad-476c-9df4-47afadef0068"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:21:26.454080 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.453859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-home" (OuterVolumeSpecName: "home") pod "aeda0c23-2dad-476c-9df4-47afadef0068" (UID: "aeda0c23-2dad-476c-9df4-47afadef0068"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:21:26.454080 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.454013    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:26.454080 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.454030    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:26.454080 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.454043    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:26.455916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.455894    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeda0c23-2dad-476c-9df4-47afadef0068-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aeda0c23-2dad-476c-9df4-47afadef0068" (UID: "aeda0c23-2dad-476c-9df4-47afadef0068"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:21:26.456023 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.455922    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-dshm" (OuterVolumeSpecName: "dshm") pod "aeda0c23-2dad-476c-9df4-47afadef0068" (UID: "aeda0c23-2dad-476c-9df4-47afadef0068"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:21:26.456023 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.455995    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeda0c23-2dad-476c-9df4-47afadef0068-kube-api-access-sctqw" (OuterVolumeSpecName: "kube-api-access-sctqw") pod "aeda0c23-2dad-476c-9df4-47afadef0068" (UID: "aeda0c23-2dad-476c-9df4-47afadef0068"). InnerVolumeSpecName "kube-api-access-sctqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:21:26.555161 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.555078    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aeda0c23-2dad-476c-9df4-47afadef0068-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:26.555161 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.555107    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aeda0c23-2dad-476c-9df4-47afadef0068-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:26.555161 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.555140    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctqw\" (UniqueName: \"kubernetes.io/projected/aeda0c23-2dad-476c-9df4-47afadef0068-kube-api-access-sctqw\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:26.855872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.855786    2579 generic.go:356] "Generic (PLEG): container finished" podID="aeda0c23-2dad-476c-9df4-47afadef0068" containerID="1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428" exitCode=0
Jul 27 17:21:26.855872 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.855847    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"
Jul 27 17:21:26.856045 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.855871    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" event={"ID":"aeda0c23-2dad-476c-9df4-47afadef0068","Type":"ContainerDied","Data":"1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"}
Jul 27 17:21:26.856045 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.855957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7" event={"ID":"aeda0c23-2dad-476c-9df4-47afadef0068","Type":"ContainerDied","Data":"e6579c56f7976f660eaff6e9aa99d656a586f632079d5bf5600541626059b1af"}
Jul 27 17:21:26.856045 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.855985    2579 scope.go:117] "RemoveContainer" containerID="1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"
Jul 27 17:21:26.865941 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.865917    2579 scope.go:117] "RemoveContainer" containerID="1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"
Jul 27 17:21:26.866261 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:21:26.866233    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428\": container with ID starting with 1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428 not found: ID does not exist" containerID="1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"
Jul 27 17:21:26.866354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.866269    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428"} err="failed to get container status \"1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428\": rpc error: code = NotFound desc = could not find container \"1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428\": container with ID starting with 1a8266f029114840a289f1ccd8a2c1424fc2ca75c517ee360f037b072ab3b428 not found: ID does not exist"
Jul 27 17:21:26.879171 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.879136    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"]
Jul 27 17:21:26.883116 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:26.883086    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-kldr7"]
Jul 27 17:21:27.892124 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:27.892088    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeda0c23-2dad-476c-9df4-47afadef0068" path="/var/lib/kubelet/pods/aeda0c23-2dad-476c-9df4-47afadef0068/volumes"
Jul 27 17:21:31.810021 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:31.809991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:21:31.810796 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:31.810776    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:21:34.715010 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:34.714973    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" podUID="77e39f35-91b0-4808-a465-0c881b026490" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:21:41.047323 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.047256    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" podUID="77e39f35-91b0-4808-a465-0c881b026490" containerName="main" containerID="cri-o://2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b" gracePeriod=45
Jul 27 17:21:41.299585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.299530    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:21:41.376160 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.376129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e39f35-91b0-4808-a465-0c881b026490-tls-certs\") pod \"77e39f35-91b0-4808-a465-0c881b026490\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") "
Jul 27 17:21:41.376331 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.376235    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdlj2\" (UniqueName: \"kubernetes.io/projected/77e39f35-91b0-4808-a465-0c881b026490-kube-api-access-jdlj2\") pod \"77e39f35-91b0-4808-a465-0c881b026490\" (UID: \"77e39f35-91b0-4808-a465-0c881b026490\") "
Jul 27 17:21:41.378252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.378218    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e39f35-91b0-4808-a465-0c881b026490-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77e39f35-91b0-4808-a465-0c881b026490" (UID: "77e39f35-91b0-4808-a465-0c881b026490"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:21:41.378362 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.378332    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e39f35-91b0-4808-a465-0c881b026490-kube-api-access-jdlj2" (OuterVolumeSpecName: "kube-api-access-jdlj2") pod "77e39f35-91b0-4808-a465-0c881b026490" (UID: "77e39f35-91b0-4808-a465-0c881b026490"). InnerVolumeSpecName "kube-api-access-jdlj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:21:41.477602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.477568    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlj2\" (UniqueName: \"kubernetes.io/projected/77e39f35-91b0-4808-a465-0c881b026490-kube-api-access-jdlj2\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:41.477602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.477596    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e39f35-91b0-4808-a465-0c881b026490-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:21:41.928343 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.928310    2579 generic.go:356] "Generic (PLEG): container finished" podID="77e39f35-91b0-4808-a465-0c881b026490" containerID="2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b" exitCode=0
Jul 27 17:21:41.928516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.928359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" event={"ID":"77e39f35-91b0-4808-a465-0c881b026490","Type":"ContainerDied","Data":"2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b"}
Jul 27 17:21:41.928516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.928394    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4" event={"ID":"77e39f35-91b0-4808-a465-0c881b026490","Type":"ContainerDied","Data":"a02c0771e5d9934289e926db7ed06141e86540f9464a17953299c6cf14eb2fe5"}
Jul 27 17:21:41.928516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.928417    2579 scope.go:117] "RemoveContainer" containerID="2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b"
Jul 27 17:21:41.928516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.928361    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"
Jul 27 17:21:41.936854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.936836    2579 scope.go:117] "RemoveContainer" containerID="2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b"
Jul 27 17:21:41.937099 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:21:41.937065    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b\": container with ID starting with 2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b not found: ID does not exist" containerID="2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b"
Jul 27 17:21:41.937209 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.937107    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b"} err="failed to get container status \"2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b\": rpc error: code = NotFound desc = could not find container \"2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b\": container with ID starting with 2bc333435f3558b2515a65f8fc49520c87c90bf85e31ae32f638be7ae308ff7b not found: ID does not exist"
Jul 27 17:21:41.947324 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.947297    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"]
Jul 27 17:21:41.952667 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:41.952643    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-7d557749p4"]
Jul 27 17:21:43.892692 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:43.892657    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e39f35-91b0-4808-a465-0c881b026490" path="/var/lib/kubelet/pods/77e39f35-91b0-4808-a465-0c881b026490/volumes"
Jul 27 17:21:48.795844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.795800    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"]
Jul 27 17:21:48.796267 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.796251    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeda0c23-2dad-476c-9df4-47afadef0068" containerName="main"
Jul 27 17:21:48.796311 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.796271    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeda0c23-2dad-476c-9df4-47afadef0068" containerName="main"
Jul 27 17:21:48.796311 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.796299    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e39f35-91b0-4808-a465-0c881b026490" containerName="main"
Jul 27 17:21:48.796311 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.796307    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e39f35-91b0-4808-a465-0c881b026490" containerName="main"
Jul 27 17:21:48.796404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.796381    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeda0c23-2dad-476c-9df4-47afadef0068" containerName="main"
Jul 27 17:21:48.796404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.796396    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e39f35-91b0-4808-a465-0c881b026490" containerName="main"
Jul 27 17:21:48.800947 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.800926    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:48.804841 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.804545    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Jul 27 17:21:48.804841 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.804581    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Jul 27 17:21:48.804841 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.804545    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 27 17:21:48.804841 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.804649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-hclt5\""
Jul 27 17:21:48.811505 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.807715    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"]
Jul 27 17:21:48.938607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.938566    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:48.938816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.938621    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8n9\" (UniqueName: \"kubernetes.io/projected/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-kube-api-access-9c8n9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:48.938816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.938727    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:48.938816 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.938781    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:48.938957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.938838    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:48.938957 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:48.938866    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.039905 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.039878    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040081 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.039917    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040081 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.039959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8n9\" (UniqueName: \"kubernetes.io/projected/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-kube-api-access-9c8n9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040081 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.039997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040081 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.040025    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040081 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.040045    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040422 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.040364    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040422 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.040412    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.040633 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.040426    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.042185 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.042157    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.042419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.042401    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.048686 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.048611    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8n9\" (UniqueName: \"kubernetes.io/projected/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-kube-api-access-9c8n9\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.112199 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.112150    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"]
Jul 27 17:21:49.115745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.115719    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:49.116070 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.116053    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.118935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.118682    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-f2zq5\""
Jul 27 17:21:49.125256 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.125215    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"]
Jul 27 17:21:49.241136 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.241099    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/62a6c058-d770-44aa-978f-785ae84b065a-kube-api-access-qdw4f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.241268 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.241146    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6c058-d770-44aa-978f-785ae84b065a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.251229 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.251203    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"]
Jul 27 17:21:49.253950 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:21:49.253922    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f668e7_f2d2_4008_a7b2_7b02589af4f6.slice/crio-5fa8196777711348d4c8313cb85c9e99b2f7be22769426fb90cae41e314d8c44 WatchSource:0}: Error finding container 5fa8196777711348d4c8313cb85c9e99b2f7be22769426fb90cae41e314d8c44: Status 404 returned error can't find the container with id 5fa8196777711348d4c8313cb85c9e99b2f7be22769426fb90cae41e314d8c44
Jul 27 17:21:49.342416 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.342384    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/62a6c058-d770-44aa-978f-785ae84b065a-kube-api-access-qdw4f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.342563 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.342438    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6c058-d770-44aa-978f-785ae84b065a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.345105 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.345076    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6c058-d770-44aa-978f-785ae84b065a-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.350123 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.350098    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/62a6c058-d770-44aa-978f-785ae84b065a-kube-api-access-qdw4f\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.428625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.428592    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.560321 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.560292    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"]
Jul 27 17:21:49.562616 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:21:49.562590    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a6c058_d770_44aa_978f_785ae84b065a.slice/crio-7c99527cd156aa7575963d8cbbf4bf91c9016d87a05d95f5593dd876675f34e2 WatchSource:0}: Error finding container 7c99527cd156aa7575963d8cbbf4bf91c9016d87a05d95f5593dd876675f34e2: Status 404 returned error can't find the container with id 7c99527cd156aa7575963d8cbbf4bf91c9016d87a05d95f5593dd876675f34e2
Jul 27 17:21:49.963731 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.963690    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" event={"ID":"62a6c058-d770-44aa-978f-785ae84b065a","Type":"ContainerStarted","Data":"6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0"}
Jul 27 17:21:49.964204 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.963752    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" event={"ID":"62a6c058-d770-44aa-978f-785ae84b065a","Type":"ContainerStarted","Data":"7c99527cd156aa7575963d8cbbf4bf91c9016d87a05d95f5593dd876675f34e2"}
Jul 27 17:21:49.964204 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.963781    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:21:49.965410 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.965388    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" event={"ID":"d3f668e7-f2d2-4008-a7b2-7b02589af4f6","Type":"ContainerStarted","Data":"a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"}
Jul 27 17:21:49.965520 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.965429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" event={"ID":"d3f668e7-f2d2-4008-a7b2-7b02589af4f6","Type":"ContainerStarted","Data":"5fa8196777711348d4c8313cb85c9e99b2f7be22769426fb90cae41e314d8c44"}
Jul 27 17:21:49.980786 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.980736    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" podStartSLOduration=0.980720668 podStartE2EDuration="980.720668ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:21:49.980007701 +0000 UTC m=+1218.565369258" watchObservedRunningTime="2026-07-27 17:21:49.980720668 +0000 UTC m=+1218.566082216"
Jul 27 17:21:49.997423 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:49.997384    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" podStartSLOduration=1.997369751 podStartE2EDuration="1.997369751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:21:49.996824851 +0000 UTC m=+1218.582186408" watchObservedRunningTime="2026-07-27 17:21:49.997369751 +0000 UTC m=+1218.582731353"
Jul 27 17:21:50.116707 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:50.116666    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:59.115938 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:59.115890    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:21:59.128568 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:21:59.128541    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:22:00.025836 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:00.025802    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:22:20.971666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:20.971636    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:22:22.495860 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.495764    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"]
Jul 27 17:22:22.499908 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.499878    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"]
Jul 27 17:22:22.506745 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:22:22.506708    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:22:22.506745 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:22:22.506745 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:22:22.506745 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de" cmd=["/bin/sleep","15"]
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.506762    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" message="" err=<
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:22:22.506776    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" podUID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" containerName="main" containerID="cri-o://a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"
Jul 27 17:22:22.507059 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.506809    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" podUID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" containerName="main" containerID="cri-o://a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de" gracePeriod=60
Jul 27 17:22:22.866056 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.866033    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:22:22.931116 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931077    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9c8n9\" (UniqueName: \"kubernetes.io/projected/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-kube-api-access-9c8n9\") pod \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") "
Jul 27 17:22:22.931298 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931141    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-model-cache\") pod \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") "
Jul 27 17:22:22.931298 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931164    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-home\") pod \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") "
Jul 27 17:22:22.931298 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931198    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tls-certs\") pod \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") "
Jul 27 17:22:22.931298 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931256    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tmp-dir\") pod \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") "
Jul 27 17:22:22.931298 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931284    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-dshm\") pod \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\" (UID: \"d3f668e7-f2d2-4008-a7b2-7b02589af4f6\") "
Jul 27 17:22:22.931588 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931407    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3f668e7-f2d2-4008-a7b2-7b02589af4f6" (UID: "d3f668e7-f2d2-4008-a7b2-7b02589af4f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:22.931588 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931419    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-home" (OuterVolumeSpecName: "home") pod "d3f668e7-f2d2-4008-a7b2-7b02589af4f6" (UID: "d3f668e7-f2d2-4008-a7b2-7b02589af4f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:22.931588 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931557    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d3f668e7-f2d2-4008-a7b2-7b02589af4f6" (UID: "d3f668e7-f2d2-4008-a7b2-7b02589af4f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:22.931949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931922    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:22.931949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931954    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:22.932119 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.931967    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:22.933834 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.933812    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d3f668e7-f2d2-4008-a7b2-7b02589af4f6" (UID: "d3f668e7-f2d2-4008-a7b2-7b02589af4f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:22:22.934067 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.934039    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-kube-api-access-9c8n9" (OuterVolumeSpecName: "kube-api-access-9c8n9") pod "d3f668e7-f2d2-4008-a7b2-7b02589af4f6" (UID: "d3f668e7-f2d2-4008-a7b2-7b02589af4f6"). InnerVolumeSpecName "kube-api-access-9c8n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:22:22.934067 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:22.934043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-dshm" (OuterVolumeSpecName: "dshm") pod "d3f668e7-f2d2-4008-a7b2-7b02589af4f6" (UID: "d3f668e7-f2d2-4008-a7b2-7b02589af4f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:22:23.032723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.032634    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:23.032723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.032665    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8n9\" (UniqueName: \"kubernetes.io/projected/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-kube-api-access-9c8n9\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:23.032723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.032678    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3f668e7-f2d2-4008-a7b2-7b02589af4f6-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:23.117378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.117341    2579 generic.go:356] "Generic (PLEG): container finished" podID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" containerID="a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de" exitCode=0
Jul 27 17:22:23.117584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.117390    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"
Jul 27 17:22:23.117584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.117421    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" event={"ID":"d3f668e7-f2d2-4008-a7b2-7b02589af4f6","Type":"ContainerDied","Data":"a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"}
Jul 27 17:22:23.117584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.117506    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7" event={"ID":"d3f668e7-f2d2-4008-a7b2-7b02589af4f6","Type":"ContainerDied","Data":"5fa8196777711348d4c8313cb85c9e99b2f7be22769426fb90cae41e314d8c44"}
Jul 27 17:22:23.117584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.117538    2579 scope.go:117] "RemoveContainer" containerID="a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"
Jul 27 17:22:23.126708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.126690    2579 scope.go:117] "RemoveContainer" containerID="a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"
Jul 27 17:22:23.127017 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:22:23.126996    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de\": container with ID starting with a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de not found: ID does not exist" containerID="a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"
Jul 27 17:22:23.127106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.127023    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de"} err="failed to get container status \"a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de\": rpc error: code = NotFound desc = could not find container \"a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de\": container with ID starting with a557e171c21869d8a92785802cf48f9e776f5de0a28c2ef28497e935d38af1de not found: ID does not exist"
Jul 27 17:22:23.138806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.138722    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"]
Jul 27 17:22:23.140717 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.140692    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-kserve-668cf8f94fcb2f7"]
Jul 27 17:22:23.892686 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:23.892652    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" path="/var/lib/kubelet/pods/d3f668e7-f2d2-4008-a7b2-7b02589af4f6/volumes"
Jul 27 17:22:30.971060 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:30.971014    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" podUID="62a6c058-d770-44aa-978f-785ae84b065a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:22:37.520255 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.520193    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" podUID="62a6c058-d770-44aa-978f-785ae84b065a" containerName="main" containerID="cri-o://6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0" gracePeriod=45
Jul 27 17:22:37.767223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.767199    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:22:37.851717 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.851632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6c058-d770-44aa-978f-785ae84b065a-tls-certs\") pod \"62a6c058-d770-44aa-978f-785ae84b065a\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") "
Jul 27 17:22:37.851873 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.851732    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/62a6c058-d770-44aa-978f-785ae84b065a-kube-api-access-qdw4f\") pod \"62a6c058-d770-44aa-978f-785ae84b065a\" (UID: \"62a6c058-d770-44aa-978f-785ae84b065a\") "
Jul 27 17:22:37.853891 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.853859    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a6c058-d770-44aa-978f-785ae84b065a-kube-api-access-qdw4f" (OuterVolumeSpecName: "kube-api-access-qdw4f") pod "62a6c058-d770-44aa-978f-785ae84b065a" (UID: "62a6c058-d770-44aa-978f-785ae84b065a"). InnerVolumeSpecName "kube-api-access-qdw4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:22:37.853891 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.853869    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a6c058-d770-44aa-978f-785ae84b065a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "62a6c058-d770-44aa-978f-785ae84b065a" (UID: "62a6c058-d770-44aa-978f-785ae84b065a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:22:37.952941 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.952908    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62a6c058-d770-44aa-978f-785ae84b065a-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:37.952941 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:37.952937    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/62a6c058-d770-44aa-978f-785ae84b065a-kube-api-access-qdw4f\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:22:38.185196 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.185097    2579 generic.go:356] "Generic (PLEG): container finished" podID="62a6c058-d770-44aa-978f-785ae84b065a" containerID="6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0" exitCode=0
Jul 27 17:22:38.185196 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.185138    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"
Jul 27 17:22:38.185196 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.185169    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" event={"ID":"62a6c058-d770-44aa-978f-785ae84b065a","Type":"ContainerDied","Data":"6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0"}
Jul 27 17:22:38.185494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.185224    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f" event={"ID":"62a6c058-d770-44aa-978f-785ae84b065a","Type":"ContainerDied","Data":"7c99527cd156aa7575963d8cbbf4bf91c9016d87a05d95f5593dd876675f34e2"}
Jul 27 17:22:38.185494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.185252    2579 scope.go:117] "RemoveContainer" containerID="6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0"
Jul 27 17:22:38.193510 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.193489    2579 scope.go:117] "RemoveContainer" containerID="6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0"
Jul 27 17:22:38.193792 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:22:38.193776    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0\": container with ID starting with 6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0 not found: ID does not exist" containerID="6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0"
Jul 27 17:22:38.193840 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.193800    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0"} err="failed to get container status \"6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0\": rpc error: code = NotFound desc = could not find container \"6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0\": container with ID starting with 6838069fe3ffee3655121a9af19b93b9108a42347b9723fb1057bc41450e78c0 not found: ID does not exist"
Jul 27 17:22:38.206946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.204272    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"]
Jul 27 17:22:38.208943 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:38.208923    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schek6j5f"]
Jul 27 17:22:39.892685 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:39.892650    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a6c058-d770-44aa-978f-785ae84b065a" path="/var/lib/kubelet/pods/62a6c058-d770-44aa-978f-785ae84b065a/volumes"
Jul 27 17:22:48.488149 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488116    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"]
Jul 27 17:22:48.488526 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488414    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a6c058-d770-44aa-978f-785ae84b065a" containerName="main"
Jul 27 17:22:48.488526 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488424    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a6c058-d770-44aa-978f-785ae84b065a" containerName="main"
Jul 27 17:22:48.488526 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488433    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" containerName="main"
Jul 27 17:22:48.488526 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488438    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" containerName="main"
Jul 27 17:22:48.488666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488526    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a6c058-d770-44aa-978f-785ae84b065a" containerName="main"
Jul 27 17:22:48.488666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.488546    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3f668e7-f2d2-4008-a7b2-7b02589af4f6" containerName="main"
Jul 27 17:22:48.492940 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.492918    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.496672 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.496649    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Jul 27 17:22:48.496775 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.496676    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Jul 27 17:22:48.496775 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.496651    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-wqghn\""
Jul 27 17:22:48.496775 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.496649    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Jul 27 17:22:48.501235 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.501210    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"]
Jul 27 17:22:48.642442 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.642413    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.642607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.642453    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.642607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.642542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215681e0-401d-4ff4-8913-f4931c760853-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.642684 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.642615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.642684 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.642665    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4925w\" (UniqueName: \"kubernetes.io/projected/215681e0-401d-4ff4-8913-f4931c760853-kube-api-access-4925w\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.642747 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.642684    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.706038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.706001    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"]
Jul 27 17:22:48.709190 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.709168    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:48.711775 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.711754    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-nctwd\""
Jul 27 17:22:48.719841 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.719820    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"]
Jul 27 17:22:48.743250 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743188    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743250 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743390 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215681e0-401d-4ff4-8913-f4931c760853-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743390 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743390 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743357    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4925w\" (UniqueName: \"kubernetes.io/projected/215681e0-401d-4ff4-8913-f4931c760853-kube-api-access-4925w\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743390 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743387    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743621 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743565    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-model-cache\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743702 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743684    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-tmp-dir\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.743740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.743707    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-home\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.745722 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.745688    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-dshm\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.745823 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.745735    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215681e0-401d-4ff4-8913-f4931c760853-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.755116 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.755095    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4925w\" (UniqueName: \"kubernetes.io/projected/215681e0-401d-4ff4-8913-f4931c760853-kube-api-access-4925w\") pod \"scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.804127 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.804101    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:48.844682 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.844641    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74be0135-7667-4e9e-8569-8f8d7599bcb8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:48.844867 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.844765    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qlvn\" (UniqueName: \"kubernetes.io/projected/74be0135-7667-4e9e-8569-8f8d7599bcb8-kube-api-access-4qlvn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:48.938005 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.937982    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"]
Jul 27 17:22:48.940625 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:22:48.940600    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215681e0_401d_4ff4_8913_f4931c760853.slice/crio-83d014b8aae739b6805b9b89bf5dee8e590d0a330c1932933804bb375042e297 WatchSource:0}: Error finding container 83d014b8aae739b6805b9b89bf5dee8e590d0a330c1932933804bb375042e297: Status 404 returned error can't find the container with id 83d014b8aae739b6805b9b89bf5dee8e590d0a330c1932933804bb375042e297
Jul 27 17:22:48.945557 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.945534    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qlvn\" (UniqueName: \"kubernetes.io/projected/74be0135-7667-4e9e-8569-8f8d7599bcb8-kube-api-access-4qlvn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:48.945643 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.945595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74be0135-7667-4e9e-8569-8f8d7599bcb8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:48.948281 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.948263    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74be0135-7667-4e9e-8569-8f8d7599bcb8-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:48.955321 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:48.955296    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qlvn\" (UniqueName: \"kubernetes.io/projected/74be0135-7667-4e9e-8569-8f8d7599bcb8-kube-api-access-4qlvn\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:49.018822 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.018795    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:49.150180 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.150154    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"]
Jul 27 17:22:49.152749 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:22:49.152720    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74be0135_7667_4e9e_8569_8f8d7599bcb8.slice/crio-cdd69038be16df345bffa97694e85fe68641ae60c0616abb3e73c3e08bfe27ae WatchSource:0}: Error finding container cdd69038be16df345bffa97694e85fe68641ae60c0616abb3e73c3e08bfe27ae: Status 404 returned error can't find the container with id cdd69038be16df345bffa97694e85fe68641ae60c0616abb3e73c3e08bfe27ae
Jul 27 17:22:49.235584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.235547    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" event={"ID":"215681e0-401d-4ff4-8913-f4931c760853","Type":"ContainerStarted","Data":"abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"}
Jul 27 17:22:49.235740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.235607    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" event={"ID":"215681e0-401d-4ff4-8913-f4931c760853","Type":"ContainerStarted","Data":"83d014b8aae739b6805b9b89bf5dee8e590d0a330c1932933804bb375042e297"}
Jul 27 17:22:49.237345 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.237323    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" event={"ID":"74be0135-7667-4e9e-8569-8f8d7599bcb8","Type":"ContainerStarted","Data":"93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8"}
Jul 27 17:22:49.237455 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.237359    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" event={"ID":"74be0135-7667-4e9e-8569-8f8d7599bcb8","Type":"ContainerStarted","Data":"cdd69038be16df345bffa97694e85fe68641ae60c0616abb3e73c3e08bfe27ae"}
Jul 27 17:22:49.237543 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.237451    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:22:49.258911 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.258808    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" podStartSLOduration=1.258790668 podStartE2EDuration="1.258790668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:22:49.258109694 +0000 UTC m=+1277.843471251" watchObservedRunningTime="2026-07-27 17:22:49.258790668 +0000 UTC m=+1277.844152279"
Jul 27 17:22:49.277379 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.277329    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" podStartSLOduration=1.277312904 podStartE2EDuration="1.277312904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:22:49.276495566 +0000 UTC m=+1277.861857120" watchObservedRunningTime="2026-07-27 17:22:49.277312904 +0000 UTC m=+1277.862674465"
Jul 27 17:22:49.805165 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:49.805125    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:58.804681 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:58.804639    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:58.817239 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:58.817212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:22:59.295548 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:22:59.295522    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:23:20.243332 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:20.243296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:23:25.626092 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:25.626052    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"]
Jul 27 17:23:25.632023 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:25.631981    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"]
Jul 27 17:23:25.641273 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:23:25.641230    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:23:25.641273 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:23:25.641273 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:23:25.641273 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100" cmd=["/bin/sleep","15"]
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:25.641287    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" message="" err=<
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:23:25.641300    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" podUID="215681e0-401d-4ff4-8913-f4931c760853" containerName="main" containerID="cri-o://abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"
Jul 27 17:23:25.641594 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:25.641335    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" podUID="215681e0-401d-4ff4-8913-f4931c760853" containerName="main" containerID="cri-o://abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100" gracePeriod=60
Jul 27 17:23:25.987393 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:25.987367    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:23:26.061568 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061537    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-tmp-dir\") pod \"215681e0-401d-4ff4-8913-f4931c760853\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") "
Jul 27 17:23:26.061745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061584    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215681e0-401d-4ff4-8913-f4931c760853-tls-certs\") pod \"215681e0-401d-4ff4-8913-f4931c760853\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") "
Jul 27 17:23:26.061745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061608    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4925w\" (UniqueName: \"kubernetes.io/projected/215681e0-401d-4ff4-8913-f4931c760853-kube-api-access-4925w\") pod \"215681e0-401d-4ff4-8913-f4931c760853\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") "
Jul 27 17:23:26.061745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061632    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-dshm\") pod \"215681e0-401d-4ff4-8913-f4931c760853\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") "
Jul 27 17:23:26.061921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061752    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-model-cache\") pod \"215681e0-401d-4ff4-8913-f4931c760853\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") "
Jul 27 17:23:26.061921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061856    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-home\") pod \"215681e0-401d-4ff4-8913-f4931c760853\" (UID: \"215681e0-401d-4ff4-8913-f4931c760853\") "
Jul 27 17:23:26.061921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061890    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "215681e0-401d-4ff4-8913-f4931c760853" (UID: "215681e0-401d-4ff4-8913-f4931c760853"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:26.061921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.061903    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-model-cache" (OuterVolumeSpecName: "model-cache") pod "215681e0-401d-4ff4-8913-f4931c760853" (UID: "215681e0-401d-4ff4-8913-f4931c760853"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:26.062146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.062126    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-home" (OuterVolumeSpecName: "home") pod "215681e0-401d-4ff4-8913-f4931c760853" (UID: "215681e0-401d-4ff4-8913-f4931c760853"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:26.062207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.062158    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:26.062207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.062174    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:26.064110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.064083    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/215681e0-401d-4ff4-8913-f4931c760853-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "215681e0-401d-4ff4-8913-f4931c760853" (UID: "215681e0-401d-4ff4-8913-f4931c760853"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:23:26.064110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.064096    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215681e0-401d-4ff4-8913-f4931c760853-kube-api-access-4925w" (OuterVolumeSpecName: "kube-api-access-4925w") pod "215681e0-401d-4ff4-8913-f4931c760853" (UID: "215681e0-401d-4ff4-8913-f4931c760853"). InnerVolumeSpecName "kube-api-access-4925w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:23:26.064110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.064100    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-dshm" (OuterVolumeSpecName: "dshm") pod "215681e0-401d-4ff4-8913-f4931c760853" (UID: "215681e0-401d-4ff4-8913-f4931c760853"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:23:26.162732 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.162647    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/215681e0-401d-4ff4-8913-f4931c760853-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:26.162732 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.162678    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4925w\" (UniqueName: \"kubernetes.io/projected/215681e0-401d-4ff4-8913-f4931c760853-kube-api-access-4925w\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:26.162732 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.162688    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:26.162732 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.162697    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/215681e0-401d-4ff4-8913-f4931c760853-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:26.404589 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.404560    2579 generic.go:356] "Generic (PLEG): container finished" podID="215681e0-401d-4ff4-8913-f4931c760853" containerID="abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100" exitCode=0
Jul 27 17:23:26.404756 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.404607    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"
Jul 27 17:23:26.404756 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.404632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" event={"ID":"215681e0-401d-4ff4-8913-f4931c760853","Type":"ContainerDied","Data":"abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"}
Jul 27 17:23:26.404756 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.404676    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9" event={"ID":"215681e0-401d-4ff4-8913-f4931c760853","Type":"ContainerDied","Data":"83d014b8aae739b6805b9b89bf5dee8e590d0a330c1932933804bb375042e297"}
Jul 27 17:23:26.404756 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.404699    2579 scope.go:117] "RemoveContainer" containerID="abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"
Jul 27 17:23:26.413191 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.413174    2579 scope.go:117] "RemoveContainer" containerID="abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"
Jul 27 17:23:26.413448 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:23:26.413429    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100\": container with ID starting with abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100 not found: ID does not exist" containerID="abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"
Jul 27 17:23:26.413545 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.413456    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100"} err="failed to get container status \"abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100\": rpc error: code = NotFound desc = could not find container \"abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100\": container with ID starting with abdca14df4b0e314063ed269933a46557af118854596d521f30b8ec39c3bc100 not found: ID does not exist"
Jul 27 17:23:26.424365 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.424338    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"]
Jul 27 17:23:26.428035 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:26.428014    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-5bff678dc7-9kpz9"]
Jul 27 17:23:27.897558 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:27.897521    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="215681e0-401d-4ff4-8913-f4931c760853" path="/var/lib/kubelet/pods/215681e0-401d-4ff4-8913-f4931c760853/volumes"
Jul 27 17:23:30.243792 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:30.243746    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:23:40.243302 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.243260    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:23:40.647810 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.647717    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerName="main" containerID="cri-o://93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8" gracePeriod=45
Jul 27 17:23:40.904692 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.904639    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:23:40.983743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.983711    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74be0135-7667-4e9e-8569-8f8d7599bcb8-tls-certs\") pod \"74be0135-7667-4e9e-8569-8f8d7599bcb8\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") "
Jul 27 17:23:40.983893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.983775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qlvn\" (UniqueName: \"kubernetes.io/projected/74be0135-7667-4e9e-8569-8f8d7599bcb8-kube-api-access-4qlvn\") pod \"74be0135-7667-4e9e-8569-8f8d7599bcb8\" (UID: \"74be0135-7667-4e9e-8569-8f8d7599bcb8\") "
Jul 27 17:23:40.985870 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.985840    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74be0135-7667-4e9e-8569-8f8d7599bcb8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74be0135-7667-4e9e-8569-8f8d7599bcb8" (UID: "74be0135-7667-4e9e-8569-8f8d7599bcb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:23:40.985968 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:40.985884    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74be0135-7667-4e9e-8569-8f8d7599bcb8-kube-api-access-4qlvn" (OuterVolumeSpecName: "kube-api-access-4qlvn") pod "74be0135-7667-4e9e-8569-8f8d7599bcb8" (UID: "74be0135-7667-4e9e-8569-8f8d7599bcb8"). InnerVolumeSpecName "kube-api-access-4qlvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:23:41.084528 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.084492    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74be0135-7667-4e9e-8569-8f8d7599bcb8-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:41.084528 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.084522    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qlvn\" (UniqueName: \"kubernetes.io/projected/74be0135-7667-4e9e-8569-8f8d7599bcb8-kube-api-access-4qlvn\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:23:41.473345 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.473310    2579 generic.go:356] "Generic (PLEG): container finished" podID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerID="93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8" exitCode=0
Jul 27 17:23:41.473806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.473361    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"
Jul 27 17:23:41.473806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.473403    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" event={"ID":"74be0135-7667-4e9e-8569-8f8d7599bcb8","Type":"ContainerDied","Data":"93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8"}
Jul 27 17:23:41.473806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.473456    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm" event={"ID":"74be0135-7667-4e9e-8569-8f8d7599bcb8","Type":"ContainerDied","Data":"cdd69038be16df345bffa97694e85fe68641ae60c0616abb3e73c3e08bfe27ae"}
Jul 27 17:23:41.473806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.473499    2579 scope.go:117] "RemoveContainer" containerID="93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8"
Jul 27 17:23:41.487248 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.487228    2579 scope.go:117] "RemoveContainer" containerID="93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8"
Jul 27 17:23:41.487569 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:23:41.487549    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8\": container with ID starting with 93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8 not found: ID does not exist" containerID="93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8"
Jul 27 17:23:41.487629 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.487578    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8"} err="failed to get container status \"93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8\": rpc error: code = NotFound desc = could not find container \"93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8\": container with ID starting with 93e59b38216226a58a1ceb884acca4c26a79c4a7a6ce74d17e338ce5dfc4ece8 not found: ID does not exist"
Jul 27 17:23:41.497319 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.497294    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"]
Jul 27 17:23:41.503038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.503017    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-69bc76qlmm"]
Jul 27 17:23:41.892487 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:41.892399    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" path="/var/lib/kubelet/pods/74be0135-7667-4e9e-8569-8f8d7599bcb8/volumes"
Jul 27 17:23:49.992167 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992131    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"]
Jul 27 17:23:49.992555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992442    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerName="main"
Jul 27 17:23:49.992555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992453    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerName="main"
Jul 27 17:23:49.992555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992494    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="215681e0-401d-4ff4-8913-f4931c760853" containerName="main"
Jul 27 17:23:49.992555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992500    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="215681e0-401d-4ff4-8913-f4931c760853" containerName="main"
Jul 27 17:23:49.992555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992551    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="74be0135-7667-4e9e-8569-8f8d7599bcb8" containerName="main"
Jul 27 17:23:49.992725 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.992563    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="215681e0-401d-4ff4-8913-f4931c760853" containerName="main"
Jul 27 17:23:49.996980 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:49.996957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.000721 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.000699    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-lctd5\""
Jul 27 17:23:50.000854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.000717    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Jul 27 17:23:50.000854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.000718    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 27 17:23:50.000854 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.000706    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Jul 27 17:23:50.005892 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.005870    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"]
Jul 27 17:23:50.055382 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.055350    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gns5l\" (UniqueName: \"kubernetes.io/projected/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-kube-api-access-gns5l\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.055571 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.055391    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.055571 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.055433    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.055571 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.055450    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.055571 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.055499    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.055571 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.055560    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.156723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.156687    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.156723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.156721    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.156980 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.156747    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.156980 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.156778    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.156980 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.156803    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gns5l\" (UniqueName: \"kubernetes.io/projected/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-kube-api-access-gns5l\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.156980 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.156822    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.157195 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.157160    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-model-cache\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.157257 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.157234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-home\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.157297 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.157264    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tmp-dir\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.159234 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.159214    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-dshm\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.159394 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.159378    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.165962 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.165940    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gns5l\" (UniqueName: \"kubernetes.io/projected/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-kube-api-access-gns5l\") pod \"scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.253330 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.253258    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"]
Jul 27 17:23:50.256821 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.256802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.259176 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.259156    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-hfh86\""
Jul 27 17:23:50.267456 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.267436    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"]
Jul 27 17:23:50.307532 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.307502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:50.358674 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.358642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b20b75-dfbf-4860-9248-59b615078c2e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.358808 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.358693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pcjn\" (UniqueName: \"kubernetes.io/projected/88b20b75-dfbf-4860-9248-59b615078c2e-kube-api-access-4pcjn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.441265 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.441237    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"]
Jul 27 17:23:50.443791 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:23:50.443754    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bb3734_e3cd_402c_a6c5_9b93abcc35a0.slice/crio-9c987225ab1d9ea8f939d1474332c639ff5594960d033deb7658951d545c31c3 WatchSource:0}: Error finding container 9c987225ab1d9ea8f939d1474332c639ff5594960d033deb7658951d545c31c3: Status 404 returned error can't find the container with id 9c987225ab1d9ea8f939d1474332c639ff5594960d033deb7658951d545c31c3
Jul 27 17:23:50.459635 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.459611    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b20b75-dfbf-4860-9248-59b615078c2e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.459752 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.459654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pcjn\" (UniqueName: \"kubernetes.io/projected/88b20b75-dfbf-4860-9248-59b615078c2e-kube-api-access-4pcjn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.462152 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.462130    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b20b75-dfbf-4860-9248-59b615078c2e-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.467788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.467767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pcjn\" (UniqueName: \"kubernetes.io/projected/88b20b75-dfbf-4860-9248-59b615078c2e-kube-api-access-4pcjn\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.515173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.515140    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" event={"ID":"97bb3734-e3cd-402c-a6c5-9b93abcc35a0","Type":"ContainerStarted","Data":"6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"}
Jul 27 17:23:50.515316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.515197    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" event={"ID":"97bb3734-e3cd-402c-a6c5-9b93abcc35a0","Type":"ContainerStarted","Data":"9c987225ab1d9ea8f939d1474332c639ff5594960d033deb7658951d545c31c3"}
Jul 27 17:23:50.532525 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.532451    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" podStartSLOduration=1.5324292229999998 podStartE2EDuration="1.532429223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:23:50.531743132 +0000 UTC m=+1339.117104688" watchObservedRunningTime="2026-07-27 17:23:50.532429223 +0000 UTC m=+1339.117790780"
Jul 27 17:23:50.567839 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.567811    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:50.707992 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:50.707969    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"]
Jul 27 17:23:50.709578 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:23:50.709548    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b20b75_dfbf_4860_9248_59b615078c2e.slice/crio-4c48e1d7bb35bfe82b79916098fb54d06c3a48319098f55c2bf2ac53c2e23c26 WatchSource:0}: Error finding container 4c48e1d7bb35bfe82b79916098fb54d06c3a48319098f55c2bf2ac53c2e23c26: Status 404 returned error can't find the container with id 4c48e1d7bb35bfe82b79916098fb54d06c3a48319098f55c2bf2ac53c2e23c26
Jul 27 17:23:51.307734 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:51.307699    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:23:51.520852 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:51.520806    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" event={"ID":"88b20b75-dfbf-4860-9248-59b615078c2e","Type":"ContainerStarted","Data":"904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"}
Jul 27 17:23:51.521046 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:51.520872    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" event={"ID":"88b20b75-dfbf-4860-9248-59b615078c2e","Type":"ContainerStarted","Data":"4c48e1d7bb35bfe82b79916098fb54d06c3a48319098f55c2bf2ac53c2e23c26"}
Jul 27 17:23:51.521046 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:51.520899    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:23:51.545193 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:23:51.545143    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" podStartSLOduration=1.545128487 podStartE2EDuration="1.545128487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:23:51.544447009 +0000 UTC m=+1340.129808565" watchObservedRunningTime="2026-07-27 17:23:51.545128487 +0000 UTC m=+1340.130490042"
Jul 27 17:24:00.308206 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:00.308128    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:24:00.320516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:00.320466    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:24:00.572770 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:00.572683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:24:22.527422 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:22.527390    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:24:23.832061 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:23.832007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"]
Jul 27 17:24:23.848393 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:23.848361    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"]
Jul 27 17:24:23.860156 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:24:23.860099    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:24:23.860156 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:24:23.860156 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:24:23.860156 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae" cmd=["/bin/sleep","15"]
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:23.860161    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" message="" err=<
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:24:23.860179    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerName="main" containerID="cri-o://6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"
Jul 27 17:24:23.860459 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:23.860218    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerName="main" containerID="cri-o://6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae" gracePeriod=60
Jul 27 17:24:24.615958 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.615935    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:24:24.652193 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652165    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-dshm\") pod \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") "
Jul 27 17:24:24.652355 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652206    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tmp-dir\") pod \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") "
Jul 27 17:24:24.652355 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652228    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tls-certs\") pod \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") "
Jul 27 17:24:24.652355 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652245    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-home\") pod \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") "
Jul 27 17:24:24.652355 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652266    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-model-cache\") pod \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") "
Jul 27 17:24:24.652355 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652332    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gns5l\" (UniqueName: \"kubernetes.io/projected/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-kube-api-access-gns5l\") pod \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\" (UID: \"97bb3734-e3cd-402c-a6c5-9b93abcc35a0\") "
Jul 27 17:24:24.652642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652543    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-home" (OuterVolumeSpecName: "home") pod "97bb3734-e3cd-402c-a6c5-9b93abcc35a0" (UID: "97bb3734-e3cd-402c-a6c5-9b93abcc35a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:24:24.652642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652558    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97bb3734-e3cd-402c-a6c5-9b93abcc35a0" (UID: "97bb3734-e3cd-402c-a6c5-9b93abcc35a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:24:24.652745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.652661    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "97bb3734-e3cd-402c-a6c5-9b93abcc35a0" (UID: "97bb3734-e3cd-402c-a6c5-9b93abcc35a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:24:24.654648 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.654618    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97bb3734-e3cd-402c-a6c5-9b93abcc35a0" (UID: "97bb3734-e3cd-402c-a6c5-9b93abcc35a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:24:24.654762 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.654712    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-kube-api-access-gns5l" (OuterVolumeSpecName: "kube-api-access-gns5l") pod "97bb3734-e3cd-402c-a6c5-9b93abcc35a0" (UID: "97bb3734-e3cd-402c-a6c5-9b93abcc35a0"). InnerVolumeSpecName "kube-api-access-gns5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:24:24.654864 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.654799    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-dshm" (OuterVolumeSpecName: "dshm") pod "97bb3734-e3cd-402c-a6c5-9b93abcc35a0" (UID: "97bb3734-e3cd-402c-a6c5-9b93abcc35a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:24:24.664404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.664380    2579 generic.go:356] "Generic (PLEG): container finished" podID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerID="6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae" exitCode=0
Jul 27 17:24:24.664547 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.664426    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"
Jul 27 17:24:24.664547 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.664453    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" event={"ID":"97bb3734-e3cd-402c-a6c5-9b93abcc35a0","Type":"ContainerDied","Data":"6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"}
Jul 27 17:24:24.664547 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.664513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" event={"ID":"97bb3734-e3cd-402c-a6c5-9b93abcc35a0","Type":"ContainerDied","Data":"9c987225ab1d9ea8f939d1474332c639ff5594960d033deb7658951d545c31c3"}
Jul 27 17:24:24.664547 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.664534    2579 scope.go:117] "RemoveContainer" containerID="6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"
Jul 27 17:24:24.674424 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.674405    2579 scope.go:117] "RemoveContainer" containerID="6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"
Jul 27 17:24:24.674758 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:24:24.674736    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae\": container with ID starting with 6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae not found: ID does not exist" containerID="6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"
Jul 27 17:24:24.674839 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.674770    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae"} err="failed to get container status \"6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae\": rpc error: code = NotFound desc = could not find container \"6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae\": container with ID starting with 6f403dfdd2cd7aa5f4749784b8e4c47bba08798b685dd1309cc7d547a9df7cae not found: ID does not exist"
Jul 27 17:24:24.686097 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.686076    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"]
Jul 27 17:24:24.692217 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.692191    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj"]
Jul 27 17:24:24.753742 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.753648    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:24.753742 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.753703    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:24.753742 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.753713    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:24.753742 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.753729    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:24.753742 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.753738    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:24.754023 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:24.753748    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gns5l\" (UniqueName: \"kubernetes.io/projected/97bb3734-e3cd-402c-a6c5-9b93abcc35a0-kube-api-access-gns5l\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:25.562004 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:25.561951    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-75747c94b5-9q2jj" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded"
Jul 27 17:24:25.894129 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:25.893634    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" path="/var/lib/kubelet/pods/97bb3734-e3cd-402c-a6c5-9b93abcc35a0/volumes"
Jul 27 17:24:32.526839 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:32.526802    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:24:35.015343 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.015304    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" message="" err="command '/bin/sleep 15' exited with 137: "
Jul 27 17:24:35.015343 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:24:35.015341    2579 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" containerName="main" containerID="cri-o://904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"
Jul 27 17:24:35.015863 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.015386    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" containerName="main" containerID="cri-o://904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15" gracePeriod=49
Jul 27 17:24:35.663279 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.663250    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6_88b20b75-dfbf-4860-9248-59b615078c2e/main/0.log"
Jul 27 17:24:35.665604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.665586    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:24:35.712765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.712739    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6_88b20b75-dfbf-4860-9248-59b615078c2e/main/0.log"
Jul 27 17:24:35.712953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.712778    2579 generic.go:356] "Generic (PLEG): container finished" podID="88b20b75-dfbf-4860-9248-59b615078c2e" containerID="904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15" exitCode=1
Jul 27 17:24:35.712953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.712810    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" event={"ID":"88b20b75-dfbf-4860-9248-59b615078c2e","Type":"ContainerDied","Data":"904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"}
Jul 27 17:24:35.712953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.712844    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"
Jul 27 17:24:35.712953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.712858    2579 scope.go:117] "RemoveContainer" containerID="904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"
Jul 27 17:24:35.712953 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.712848    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6" event={"ID":"88b20b75-dfbf-4860-9248-59b615078c2e","Type":"ContainerDied","Data":"4c48e1d7bb35bfe82b79916098fb54d06c3a48319098f55c2bf2ac53c2e23c26"}
Jul 27 17:24:35.721062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.721043    2579 scope.go:117] "RemoveContainer" containerID="904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"
Jul 27 17:24:35.721317 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:24:35.721298    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15\": container with ID starting with 904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15 not found: ID does not exist" containerID="904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"
Jul 27 17:24:35.721361 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.721325    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15"} err="failed to get container status \"904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15\": rpc error: code = NotFound desc = could not find container \"904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15\": container with ID starting with 904f1997cb57ebadd8d50c7b5779a284d4731d7f2f7717b4cacb525b2a966f15 not found: ID does not exist"
Jul 27 17:24:35.742769 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.742739    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pcjn\" (UniqueName: \"kubernetes.io/projected/88b20b75-dfbf-4860-9248-59b615078c2e-kube-api-access-4pcjn\") pod \"88b20b75-dfbf-4860-9248-59b615078c2e\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") "
Jul 27 17:24:35.742893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.742808    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b20b75-dfbf-4860-9248-59b615078c2e-tls-certs\") pod \"88b20b75-dfbf-4860-9248-59b615078c2e\" (UID: \"88b20b75-dfbf-4860-9248-59b615078c2e\") "
Jul 27 17:24:35.744902 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.744877    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b20b75-dfbf-4860-9248-59b615078c2e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88b20b75-dfbf-4860-9248-59b615078c2e" (UID: "88b20b75-dfbf-4860-9248-59b615078c2e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:24:35.745007 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.744970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b20b75-dfbf-4860-9248-59b615078c2e-kube-api-access-4pcjn" (OuterVolumeSpecName: "kube-api-access-4pcjn") pod "88b20b75-dfbf-4860-9248-59b615078c2e" (UID: "88b20b75-dfbf-4860-9248-59b615078c2e"). InnerVolumeSpecName "kube-api-access-4pcjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:24:35.843634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.843534    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88b20b75-dfbf-4860-9248-59b615078c2e-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:35.843634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:35.843577    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pcjn\" (UniqueName: \"kubernetes.io/projected/88b20b75-dfbf-4860-9248-59b615078c2e-kube-api-access-4pcjn\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:24:36.028572 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:36.028540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"]
Jul 27 17:24:36.033959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:36.033934    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-5c48678gd6c6"]
Jul 27 17:24:37.893162 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:37.893124    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" path="/var/lib/kubelet/pods/88b20b75-dfbf-4860-9248-59b615078c2e/volumes"
Jul 27 17:24:54.791106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791074    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"]
Jul 27 17:24:54.791535 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791392    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" containerName="main"
Jul 27 17:24:54.791535 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791404    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" containerName="main"
Jul 27 17:24:54.791535 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791422    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerName="main"
Jul 27 17:24:54.791535 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791427    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerName="main"
Jul 27 17:24:54.791535 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791500    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="88b20b75-dfbf-4860-9248-59b615078c2e" containerName="main"
Jul 27 17:24:54.791535 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.791509    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bb3734-e3cd-402c-a6c5-9b93abcc35a0" containerName="main"
Jul 27 17:24:54.795753 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.795734    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.799303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.799287    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Jul 27 17:24:54.799417 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.799289    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-spbz2\""
Jul 27 17:24:54.799417 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.799371    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 27 17:24:54.799566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.799419    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Jul 27 17:24:54.804634 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.804614    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"]
Jul 27 17:24:54.898760 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.898730    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.898934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.898779    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.898934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.898798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641e9c16-2bad-47f6-a61e-f2550c320a17-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.898934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.898821    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.898934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.898873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.898934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.898901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl9kw\" (UniqueName: \"kubernetes.io/projected/641e9c16-2bad-47f6-a61e-f2550c320a17-kube-api-access-rl9kw\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.999948 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.999911    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:54.999948 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.999948    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641e9c16-2bad-47f6-a61e-f2550c320a17-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000198 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:54.999971    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000198 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.000010    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000198 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.000125    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl9kw\" (UniqueName: \"kubernetes.io/projected/641e9c16-2bad-47f6-a61e-f2550c320a17-kube-api-access-rl9kw\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000351 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.000210    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000447 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.000429    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-home\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000588 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.000564    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-model-cache\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.000690 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.000569    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-tmp-dir\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.002893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.002871    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-dshm\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.003131 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.003113    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641e9c16-2bad-47f6-a61e-f2550c320a17-tls-certs\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.011178 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.011148    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl9kw\" (UniqueName: \"kubernetes.io/projected/641e9c16-2bad-47f6-a61e-f2550c320a17-kube-api-access-rl9kw\") pod \"scheduler-custom-template-test-kserve-86c4fb7489-t28md\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.039935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.039906    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"]
Jul 27 17:24:55.043385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.043343    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.045917 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.045895    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-jdtv5\""
Jul 27 17:24:55.051862 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.051836    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"]
Jul 27 17:24:55.100658 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.100623    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.100834 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.100678    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c14ec6-fa28-471c-9bff-f703c52aa663-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.100834 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.100698    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xnl\" (UniqueName: \"kubernetes.io/projected/b7c14ec6-fa28-471c-9bff-f703c52aa663-kube-api-access-z7xnl\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.100834 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.100795    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.106599 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.106578    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:24:55.201262 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.201225    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.201415 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.201296    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.201415 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.201368    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c14ec6-fa28-471c-9bff-f703c52aa663-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.201415 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.201396    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xnl\" (UniqueName: \"kubernetes.io/projected/b7c14ec6-fa28-471c-9bff-f703c52aa663-kube-api-access-z7xnl\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.201692 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.201670    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.201763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.201733    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.204207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.204182    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c14ec6-fa28-471c-9bff-f703c52aa663-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.210373 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.210352    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xnl\" (UniqueName: \"kubernetes.io/projected/b7c14ec6-fa28-471c-9bff-f703c52aa663-kube-api-access-z7xnl\") pod \"scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.242059 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.242035    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"]
Jul 27 17:24:55.243523 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:24:55.243492    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641e9c16_2bad_47f6_a61e_f2550c320a17.slice/crio-93d833d7a8daee52a4c1295b521e1e46bb7bf30a0946d335fff8dd85afd9e033 WatchSource:0}: Error finding container 93d833d7a8daee52a4c1295b521e1e46bb7bf30a0946d335fff8dd85afd9e033: Status 404 returned error can't find the container with id 93d833d7a8daee52a4c1295b521e1e46bb7bf30a0946d335fff8dd85afd9e033
Jul 27 17:24:55.354048 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.354014    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.485973 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.485940    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"]
Jul 27 17:24:55.488738 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:24:55.488710    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7c14ec6_fa28_471c_9bff_f703c52aa663.slice/crio-ba184981d54f04dbb0ce6afff18bc363b1def92706c9383120b12856b88eebd3 WatchSource:0}: Error finding container ba184981d54f04dbb0ce6afff18bc363b1def92706c9383120b12856b88eebd3: Status 404 returned error can't find the container with id ba184981d54f04dbb0ce6afff18bc363b1def92706c9383120b12856b88eebd3
Jul 27 17:24:55.799346 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.799249    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" event={"ID":"641e9c16-2bad-47f6-a61e-f2550c320a17","Type":"ContainerStarted","Data":"577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"}
Jul 27 17:24:55.799346 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.799321    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" event={"ID":"641e9c16-2bad-47f6-a61e-f2550c320a17","Type":"ContainerStarted","Data":"93d833d7a8daee52a4c1295b521e1e46bb7bf30a0946d335fff8dd85afd9e033"}
Jul 27 17:24:55.800687 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.800663    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" event={"ID":"b7c14ec6-fa28-471c-9bff-f703c52aa663","Type":"ContainerStarted","Data":"887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352"}
Jul 27 17:24:55.800806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.800705    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" event={"ID":"b7c14ec6-fa28-471c-9bff-f703c52aa663","Type":"ContainerStarted","Data":"ba184981d54f04dbb0ce6afff18bc363b1def92706c9383120b12856b88eebd3"}
Jul 27 17:24:55.800806 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.800761    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:24:55.816437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.816399    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" podStartSLOduration=1.816384694 podStartE2EDuration="1.816384694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:24:55.814960004 +0000 UTC m=+1404.400321576" watchObservedRunningTime="2026-07-27 17:24:55.816384694 +0000 UTC m=+1404.401746251"
Jul 27 17:24:55.831081 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:55.831033    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" podStartSLOduration=0.831015606 podStartE2EDuration="831.015606ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:24:55.830378455 +0000 UTC m=+1404.415740022" watchObservedRunningTime="2026-07-27 17:24:55.831015606 +0000 UTC m=+1404.416377162"
Jul 27 17:24:56.107110 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:24:56.107025    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:25:05.106937 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:05.106905    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:25:05.119494 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:05.119454    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:25:05.854318 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:05.854286    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:25:26.807268 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:26.807188    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:25:28.287128 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.287094    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"]
Jul 27 17:25:28.293287 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.293255    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"]
Jul 27 17:25:28.297611 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:25:28.297576    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:25:28.297611 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:25:28.297611 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:25:28.297611 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf" cmd=["/bin/sleep","15"]
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.297626    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" message="" err=<
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:25:28.297642    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" podUID="641e9c16-2bad-47f6-a61e-f2550c320a17" containerName="main" containerID="cri-o://577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"
Jul 27 17:25:28.297868 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.297682    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" podUID="641e9c16-2bad-47f6-a61e-f2550c320a17" containerName="main" containerID="cri-o://577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf" gracePeriod=60
Jul 27 17:25:28.655165 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.655137    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:25:28.682766 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.682736    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-home\") pod \"641e9c16-2bad-47f6-a61e-f2550c320a17\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") "
Jul 27 17:25:28.682955 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.682775    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641e9c16-2bad-47f6-a61e-f2550c320a17-tls-certs\") pod \"641e9c16-2bad-47f6-a61e-f2550c320a17\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") "
Jul 27 17:25:28.682955 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.682801    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-model-cache\") pod \"641e9c16-2bad-47f6-a61e-f2550c320a17\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") "
Jul 27 17:25:28.682955 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.682835    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rl9kw\" (UniqueName: \"kubernetes.io/projected/641e9c16-2bad-47f6-a61e-f2550c320a17-kube-api-access-rl9kw\") pod \"641e9c16-2bad-47f6-a61e-f2550c320a17\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") "
Jul 27 17:25:28.682955 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.682861    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-tmp-dir\") pod \"641e9c16-2bad-47f6-a61e-f2550c320a17\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") "
Jul 27 17:25:28.682955 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.682915    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-dshm\") pod \"641e9c16-2bad-47f6-a61e-f2550c320a17\" (UID: \"641e9c16-2bad-47f6-a61e-f2550c320a17\") "
Jul 27 17:25:28.683247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.683056    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-home" (OuterVolumeSpecName: "home") pod "641e9c16-2bad-47f6-a61e-f2550c320a17" (UID: "641e9c16-2bad-47f6-a61e-f2550c320a17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:28.683247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.683140    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-model-cache" (OuterVolumeSpecName: "model-cache") pod "641e9c16-2bad-47f6-a61e-f2550c320a17" (UID: "641e9c16-2bad-47f6-a61e-f2550c320a17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:28.683247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.683181    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:28.683247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.683177    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "641e9c16-2bad-47f6-a61e-f2550c320a17" (UID: "641e9c16-2bad-47f6-a61e-f2550c320a17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:28.685210 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.685181    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-dshm" (OuterVolumeSpecName: "dshm") pod "641e9c16-2bad-47f6-a61e-f2550c320a17" (UID: "641e9c16-2bad-47f6-a61e-f2550c320a17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:28.685345 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.685258    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641e9c16-2bad-47f6-a61e-f2550c320a17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "641e9c16-2bad-47f6-a61e-f2550c320a17" (UID: "641e9c16-2bad-47f6-a61e-f2550c320a17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:25:28.685345 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.685277    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641e9c16-2bad-47f6-a61e-f2550c320a17-kube-api-access-rl9kw" (OuterVolumeSpecName: "kube-api-access-rl9kw") pod "641e9c16-2bad-47f6-a61e-f2550c320a17" (UID: "641e9c16-2bad-47f6-a61e-f2550c320a17"). InnerVolumeSpecName "kube-api-access-rl9kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:25:28.783951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.783910    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:28.783951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.783944    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:28.783951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.783953    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/641e9c16-2bad-47f6-a61e-f2550c320a17-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:28.783951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.783961    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/641e9c16-2bad-47f6-a61e-f2550c320a17-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:28.784219 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.783969    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rl9kw\" (UniqueName: \"kubernetes.io/projected/641e9c16-2bad-47f6-a61e-f2550c320a17-kube-api-access-rl9kw\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:28.943010 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.942919    2579 generic.go:356] "Generic (PLEG): container finished" podID="641e9c16-2bad-47f6-a61e-f2550c320a17" containerID="577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf" exitCode=0
Jul 27 17:25:28.943010 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.942965    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" event={"ID":"641e9c16-2bad-47f6-a61e-f2550c320a17","Type":"ContainerDied","Data":"577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"}
Jul 27 17:25:28.943010 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.943003    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md" event={"ID":"641e9c16-2bad-47f6-a61e-f2550c320a17","Type":"ContainerDied","Data":"93d833d7a8daee52a4c1295b521e1e46bb7bf30a0946d335fff8dd85afd9e033"}
Jul 27 17:25:28.943239 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.943025    2579 scope.go:117] "RemoveContainer" containerID="577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"
Jul 27 17:25:28.943239 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.942967    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"
Jul 27 17:25:28.952769 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.952749    2579 scope.go:117] "RemoveContainer" containerID="577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"
Jul 27 17:25:28.953078 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:25:28.953059    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf\": container with ID starting with 577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf not found: ID does not exist" containerID="577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"
Jul 27 17:25:28.953149 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.953089    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf"} err="failed to get container status \"577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf\": rpc error: code = NotFound desc = could not find container \"577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf\": container with ID starting with 577fbc2a4adaa3cd0d5b96f94595a800198239df0c879566c050713cf1654eaf not found: ID does not exist"
Jul 27 17:25:28.973704 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.973673    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"]
Jul 27 17:25:28.974935 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:28.974912    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-86c4fb7489-t28md"]
Jul 27 17:25:29.892767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:29.892732    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="641e9c16-2bad-47f6-a61e-f2550c320a17" path="/var/lib/kubelet/pods/641e9c16-2bad-47f6-a61e-f2550c320a17/volumes"
Jul 27 17:25:36.807170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:36.807130    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" podUID="b7c14ec6-fa28-471c-9bff-f703c52aa663" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:25:43.314013 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.313945    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" podUID="b7c14ec6-fa28-471c-9bff-f703c52aa663" containerName="main" containerID="cri-o://887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352" gracePeriod=45
Jul 27 17:25:43.567348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.567276    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:25:43.608216 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.608178    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-tokenizer-cache\") pod \"b7c14ec6-fa28-471c-9bff-f703c52aa663\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") "
Jul 27 17:25:43.608404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.608227    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-cachi2-cache\") pod \"b7c14ec6-fa28-471c-9bff-f703c52aa663\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") "
Jul 27 17:25:43.608404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.608269    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c14ec6-fa28-471c-9bff-f703c52aa663-tls-certs\") pod \"b7c14ec6-fa28-471c-9bff-f703c52aa663\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") "
Jul 27 17:25:43.608404 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.608325    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7xnl\" (UniqueName: \"kubernetes.io/projected/b7c14ec6-fa28-471c-9bff-f703c52aa663-kube-api-access-z7xnl\") pod \"b7c14ec6-fa28-471c-9bff-f703c52aa663\" (UID: \"b7c14ec6-fa28-471c-9bff-f703c52aa663\") "
Jul 27 17:25:43.608604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.608534    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "b7c14ec6-fa28-471c-9bff-f703c52aa663" (UID: "b7c14ec6-fa28-471c-9bff-f703c52aa663"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:43.608648 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.608601    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "b7c14ec6-fa28-471c-9bff-f703c52aa663" (UID: "b7c14ec6-fa28-471c-9bff-f703c52aa663"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:25:43.610784 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.610759    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7c14ec6-fa28-471c-9bff-f703c52aa663-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7c14ec6-fa28-471c-9bff-f703c52aa663" (UID: "b7c14ec6-fa28-471c-9bff-f703c52aa663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:25:43.610874 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.610762    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7c14ec6-fa28-471c-9bff-f703c52aa663-kube-api-access-z7xnl" (OuterVolumeSpecName: "kube-api-access-z7xnl") pod "b7c14ec6-fa28-471c-9bff-f703c52aa663" (UID: "b7c14ec6-fa28-471c-9bff-f703c52aa663"). InnerVolumeSpecName "kube-api-access-z7xnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:25:43.709825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.709787    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xnl\" (UniqueName: \"kubernetes.io/projected/b7c14ec6-fa28-471c-9bff-f703c52aa663-kube-api-access-z7xnl\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:43.709825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.709817    2579 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-tokenizer-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:43.709825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.709827    2579 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7c14ec6-fa28-471c-9bff-f703c52aa663-cachi2-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:43.709825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:43.709836    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7c14ec6-fa28-471c-9bff-f703c52aa663-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:25:44.014014 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.013973    2579 generic.go:356] "Generic (PLEG): container finished" podID="b7c14ec6-fa28-471c-9bff-f703c52aa663" containerID="887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352" exitCode=0
Jul 27 17:25:44.014182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.014035    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"
Jul 27 17:25:44.014182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.014056    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" event={"ID":"b7c14ec6-fa28-471c-9bff-f703c52aa663","Type":"ContainerDied","Data":"887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352"}
Jul 27 17:25:44.014182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.014116    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx" event={"ID":"b7c14ec6-fa28-471c-9bff-f703c52aa663","Type":"ContainerDied","Data":"ba184981d54f04dbb0ce6afff18bc363b1def92706c9383120b12856b88eebd3"}
Jul 27 17:25:44.014182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.014139    2579 scope.go:117] "RemoveContainer" containerID="887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352"
Jul 27 17:25:44.024004 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.023979    2579 scope.go:117] "RemoveContainer" containerID="887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352"
Jul 27 17:25:44.024313 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:25:44.024294    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352\": container with ID starting with 887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352 not found: ID does not exist" containerID="887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352"
Jul 27 17:25:44.024397 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.024319    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352"} err="failed to get container status \"887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352\": rpc error: code = NotFound desc = could not find container \"887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352\": container with ID starting with 887ed6ea989ab5b2e7a859e02a0548a320dec53bcd2eeff8796336ce63961352 not found: ID does not exist"
Jul 27 17:25:44.030543 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.030513    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"]
Jul 27 17:25:44.034649 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:44.034607    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-6c7jgxrx"]
Jul 27 17:25:45.892595 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:45.892558    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7c14ec6-fa28-471c-9bff-f703c52aa663" path="/var/lib/kubelet/pods/b7c14ec6-fa28-471c-9bff-f703c52aa663/volumes"
Jul 27 17:25:55.817492 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817433    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"]
Jul 27 17:25:55.818026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817795    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7c14ec6-fa28-471c-9bff-f703c52aa663" containerName="main"
Jul 27 17:25:55.818026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817809    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7c14ec6-fa28-471c-9bff-f703c52aa663" containerName="main"
Jul 27 17:25:55.818026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817825    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="641e9c16-2bad-47f6-a61e-f2550c320a17" containerName="main"
Jul 27 17:25:55.818026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817830    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="641e9c16-2bad-47f6-a61e-f2550c320a17" containerName="main"
Jul 27 17:25:55.818026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817882    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7c14ec6-fa28-471c-9bff-f703c52aa663" containerName="main"
Jul 27 17:25:55.818026 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.817892    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="641e9c16-2bad-47f6-a61e-f2550c320a17" containerName="main"
Jul 27 17:25:55.822222 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.822195    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:55.825853 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.825827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-gn825\""
Jul 27 17:25:55.825982 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.825827    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-dockercfg-skngj\""
Jul 27 17:25:55.825982 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.825836    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Jul 27 17:25:55.825982 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.825837    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Jul 27 17:25:55.825982 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.825836    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 27 17:25:55.834575 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.834547    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"]
Jul 27 17:25:55.910688 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.910652    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9msgb\" (UniqueName: \"kubernetes.io/projected/e6477d40-f2f0-4a39-92ce-3223ed998bc1-kube-api-access-9msgb\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:55.910688 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.910690    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:55.910905 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.910793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:55.910905 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.910825    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:55.910905 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.910878    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:55.910905 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:55.910901    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012297 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012256    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012306    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012336    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9msgb\" (UniqueName: \"kubernetes.io/projected/e6477d40-f2f0-4a39-92ce-3223ed998bc1-kube-api-access-9msgb\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012552 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012363    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012715 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012543    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012715 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012592    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012805 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012770    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tmp-dir\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012848 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012832    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-home\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.012914 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.012898    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-model-cache\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.014851 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.014831    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-dshm\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.015089 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.015072    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.025020 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.024977    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9msgb\" (UniqueName: \"kubernetes.io/projected/e6477d40-f2f0-4a39-92ce-3223ed998bc1-kube-api-access-9msgb\") pod \"scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.132968 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.132885    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:56.271431 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.271404    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"]
Jul 27 17:25:56.274022 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:25:56.273996    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6477d40_f2f0_4a39_92ce_3223ed998bc1.slice/crio-d4047713c6cbb48d56e677ba8a83e022c86c7de8a06d9aab7948b18cae5d03f8 WatchSource:0}: Error finding container d4047713c6cbb48d56e677ba8a83e022c86c7de8a06d9aab7948b18cae5d03f8: Status 404 returned error can't find the container with id d4047713c6cbb48d56e677ba8a83e022c86c7de8a06d9aab7948b18cae5d03f8
Jul 27 17:25:56.275942 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:56.275921    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 27 17:25:57.071123 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:57.071082    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" event={"ID":"e6477d40-f2f0-4a39-92ce-3223ed998bc1","Type":"ContainerStarted","Data":"d4047713c6cbb48d56e677ba8a83e022c86c7de8a06d9aab7948b18cae5d03f8"}
Jul 27 17:25:58.076825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:58.076784    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" event={"ID":"e6477d40-f2f0-4a39-92ce-3223ed998bc1","Type":"ContainerStarted","Data":"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"}
Jul 27 17:25:58.077321 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:58.076861    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:25:59.083124 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:59.083088    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" event={"ID":"e6477d40-f2f0-4a39-92ce-3223ed998bc1","Type":"ContainerStarted","Data":"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"}
Jul 27 17:25:59.102636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:59.102582    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" podStartSLOduration=2.692165673 podStartE2EDuration="4.102566519s" totalImagesPullingTime="1.410400846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:25:59.101685757 +0000 UTC m=+1467.687047520" watchObservedRunningTime="2026-07-27 17:25:59.102566519 +0000 UTC m=+1467.687928090"
Jul 27 17:25:59.134002 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:25:59.133962    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:26:06.134027 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:06.133990    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:26:06.146662 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:06.146635    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:26:07.128323 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:07.128296    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:26:07.129642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:07.129617    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:26:28.781340 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.781294    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"]
Jul 27 17:26:28.784577 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.784515    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.788152 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.788128    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"default-dockercfg-2cwvx\""
Jul 27 17:26:28.788152 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.788143    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"autoscale-hpa-deploy-kserve-self-signed-certs\""
Jul 27 17:26:28.788335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.788129    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"openshift-service-ca.crt\""
Jul 27 17:26:28.788335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.788130    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4\"/\"kube-root-ca.crt\""
Jul 27 17:26:28.789072 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.789051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.789183 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.789165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlf2v\" (UniqueName: \"kubernetes.io/projected/50914253-8935-4351-a8f8-eb12fe2cc6e4-kube-api-access-vlf2v\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.789227 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.789209    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.789265 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.789239    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50914253-8935-4351-a8f8-eb12fe2cc6e4-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.789315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.789264    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.789315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.789303    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.795544 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.795521    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"]
Jul 27 17:26:28.890071 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890040    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlf2v\" (UniqueName: \"kubernetes.io/projected/50914253-8935-4351-a8f8-eb12fe2cc6e4-kube-api-access-vlf2v\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890071 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890076    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50914253-8935-4351-a8f8-eb12fe2cc6e4-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890117    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890135    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890265    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890605 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890517    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-home\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890605 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890585    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-tmp-dir\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.890605 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.890600    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-model-cache\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.892673 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.892645    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-dshm\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.892835 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.892817    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50914253-8935-4351-a8f8-eb12fe2cc6e4-tls-certs\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:28.898924 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:28.898897    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlf2v\" (UniqueName: \"kubernetes.io/projected/50914253-8935-4351-a8f8-eb12fe2cc6e4-kube-api-access-vlf2v\") pod \"autoscale-hpa-deploy-kserve-66746bd8b9-crcfs\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") " pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:29.095388 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:29.095293    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:29.236560 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:29.236531    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"]
Jul 27 17:26:29.239535 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:26:29.239506    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50914253_8935_4351_a8f8_eb12fe2cc6e4.slice/crio-a84fdf4a6b8a8bc049a17835b614506fd175a0b1fa22387557f4b9f3c5abbc55 WatchSource:0}: Error finding container a84fdf4a6b8a8bc049a17835b614506fd175a0b1fa22387557f4b9f3c5abbc55: Status 404 returned error can't find the container with id a84fdf4a6b8a8bc049a17835b614506fd175a0b1fa22387557f4b9f3c5abbc55
Jul 27 17:26:30.217986 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:30.217942    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" event={"ID":"50914253-8935-4351-a8f8-eb12fe2cc6e4","Type":"ContainerStarted","Data":"1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"}
Jul 27 17:26:30.218398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:30.218008    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" event={"ID":"50914253-8935-4351-a8f8-eb12fe2cc6e4","Type":"ContainerStarted","Data":"a84fdf4a6b8a8bc049a17835b614506fd175a0b1fa22387557f4b9f3c5abbc55"}
Jul 27 17:26:30.238374 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:30.238320    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" podStartSLOduration=2.238303798 podStartE2EDuration="2.238303798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:26:30.236732644 +0000 UTC m=+1498.822094212" watchObservedRunningTime="2026-07-27 17:26:30.238303798 +0000 UTC m=+1498.823665354"
Jul 27 17:26:31.096005 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:31.095966    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:31.837284 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:31.837248    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:26:31.843382 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:31.843356    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:26:39.095822 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:39.095788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:39.108301 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:39.108269    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:26:39.272666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:26:39.272638    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:27:35.515272 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:35.515237    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"]
Jul 27 17:27:35.524227 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:27:35.524184    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:27:35.524227 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:27:35.524227 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:27:35.524227 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68" cmd=["/bin/sleep","15"]
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:35.524251    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" message="" err=<
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:27:35.524270    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="main" containerID="cri-o://8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"
Jul 27 17:27:35.524464 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:35.524327    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="main" containerID="cri-o://8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68" gracePeriod=60
Jul 27 17:27:35.636985 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:35.636954    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="llm-d-routing-sidecar" containerID="cri-o://bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5" gracePeriod=60
Jul 27 17:27:35.893423 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:35.893393    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:27:36.022047 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022015    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9msgb\" (UniqueName: \"kubernetes.io/projected/e6477d40-f2f0-4a39-92ce-3223ed998bc1-kube-api-access-9msgb\") pod \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") "
Jul 27 17:27:36.022207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022063    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tls-certs\") pod \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") "
Jul 27 17:27:36.022207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022105    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tmp-dir\") pod \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") "
Jul 27 17:27:36.022207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022132    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-home\") pod \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") "
Jul 27 17:27:36.022207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022169    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-dshm\") pod \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") "
Jul 27 17:27:36.022207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022204    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-model-cache\") pod \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\" (UID: \"e6477d40-f2f0-4a39-92ce-3223ed998bc1\") "
Jul 27 17:27:36.022459 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022425    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6477d40-f2f0-4a39-92ce-3223ed998bc1" (UID: "e6477d40-f2f0-4a39-92ce-3223ed998bc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.022554 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022433    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-home" (OuterVolumeSpecName: "home") pod "e6477d40-f2f0-4a39-92ce-3223ed998bc1" (UID: "e6477d40-f2f0-4a39-92ce-3223ed998bc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.022605 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.022539    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6477d40-f2f0-4a39-92ce-3223ed998bc1" (UID: "e6477d40-f2f0-4a39-92ce-3223ed998bc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.024437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.024412    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6477d40-f2f0-4a39-92ce-3223ed998bc1" (UID: "e6477d40-f2f0-4a39-92ce-3223ed998bc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:27:36.024437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.024423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6477d40-f2f0-4a39-92ce-3223ed998bc1-kube-api-access-9msgb" (OuterVolumeSpecName: "kube-api-access-9msgb") pod "e6477d40-f2f0-4a39-92ce-3223ed998bc1" (UID: "e6477d40-f2f0-4a39-92ce-3223ed998bc1"). InnerVolumeSpecName "kube-api-access-9msgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:27:36.024569 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.024493    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-dshm" (OuterVolumeSpecName: "dshm") pod "e6477d40-f2f0-4a39-92ce-3223ed998bc1" (UID: "e6477d40-f2f0-4a39-92ce-3223ed998bc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:27:36.123159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.123078    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.123159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.123106    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.123159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.123114    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.123159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.123122    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.123159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.123129    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6477d40-f2f0-4a39-92ce-3223ed998bc1-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.123159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.123137    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9msgb\" (UniqueName: \"kubernetes.io/projected/e6477d40-f2f0-4a39-92ce-3223ed998bc1-kube-api-access-9msgb\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:27:36.512623 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512581    2579 generic.go:356] "Generic (PLEG): container finished" podID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerID="8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68" exitCode=0
Jul 27 17:27:36.512623 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512602    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" event={"ID":"e6477d40-f2f0-4a39-92ce-3223ed998bc1","Type":"ContainerDied","Data":"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"}
Jul 27 17:27:36.512623 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512619    2579 generic.go:356] "Generic (PLEG): container finished" podID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerID="bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5" exitCode=0
Jul 27 17:27:36.512623 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512631    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"
Jul 27 17:27:36.512969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512667    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" event={"ID":"e6477d40-f2f0-4a39-92ce-3223ed998bc1","Type":"ContainerDied","Data":"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"}
Jul 27 17:27:36.512969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512692    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm" event={"ID":"e6477d40-f2f0-4a39-92ce-3223ed998bc1","Type":"ContainerDied","Data":"d4047713c6cbb48d56e677ba8a83e022c86c7de8a06d9aab7948b18cae5d03f8"}
Jul 27 17:27:36.512969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.512718    2579 scope.go:117] "RemoveContainer" containerID="8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"
Jul 27 17:27:36.521433 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.521418    2579 scope.go:117] "RemoveContainer" containerID="bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"
Jul 27 17:27:36.528968 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.528793    2579 scope.go:117] "RemoveContainer" containerID="8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"
Jul 27 17:27:36.529097 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:27:36.529070    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68\": container with ID starting with 8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68 not found: ID does not exist" containerID="8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"
Jul 27 17:27:36.529217 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.529105    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"} err="failed to get container status \"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68\": rpc error: code = NotFound desc = could not find container \"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68\": container with ID starting with 8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68 not found: ID does not exist"
Jul 27 17:27:36.529217 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.529132    2579 scope.go:117] "RemoveContainer" containerID="bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"
Jul 27 17:27:36.529399 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:27:36.529381    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5\": container with ID starting with bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5 not found: ID does not exist" containerID="bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"
Jul 27 17:27:36.529435 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.529406    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"} err="failed to get container status \"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5\": rpc error: code = NotFound desc = could not find container \"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5\": container with ID starting with bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5 not found: ID does not exist"
Jul 27 17:27:36.529435 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.529424    2579 scope.go:117] "RemoveContainer" containerID="8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"
Jul 27 17:27:36.529798 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.529776    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68"} err="failed to get container status \"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68\": rpc error: code = NotFound desc = could not find container \"8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68\": container with ID starting with 8bcfae04ea2f5bbde32815ae3afdd530040edde9d35a8c7df8baefb9e24d4c68 not found: ID does not exist"
Jul 27 17:27:36.529798 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.529798    2579 scope.go:117] "RemoveContainer" containerID="bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"
Jul 27 17:27:36.530023 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.530007    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5"} err="failed to get container status \"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5\": rpc error: code = NotFound desc = could not find container \"bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5\": container with ID starting with bc49c584d7bf659a7466d74600db034b08964c4edaf2e5ef23c24042615026f5 not found: ID does not exist"
Jul 27 17:27:36.548878 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.548853    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"]
Jul 27 17:27:36.558457 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:36.558435    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-5648d94946-fghlm"]
Jul 27 17:27:37.892776 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:27:37.892741    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" path="/var/lib/kubelet/pods/e6477d40-f2f0-4a39-92ce-3223ed998bc1/volumes"
Jul 27 17:28:01.035809 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.035769    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"]
Jul 27 17:28:01.036192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.036073    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="main"
Jul 27 17:28:01.036192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.036083    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="main"
Jul 27 17:28:01.036192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.036094    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="llm-d-routing-sidecar"
Jul 27 17:28:01.036192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.036100    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="llm-d-routing-sidecar"
Jul 27 17:28:01.036192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.036159    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="main"
Jul 27 17:28:01.036192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.036167    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6477d40-f2f0-4a39-92ce-3223ed998bc1" containerName="llm-d-routing-sidecar"
Jul 27 17:28:01.040856 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.040834    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.043443 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.043421    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-zc9jr\""
Jul 27 17:28:01.044460 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.044437    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-95l2n\""
Jul 27 17:28:01.044460 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.044454    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Jul 27 17:28:01.044641 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.044454    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 27 17:28:01.044641 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.044524    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Jul 27 17:28:01.054577 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.054555    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"]
Jul 27 17:28:01.118920 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.118882    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.119093 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.118925    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.119093 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.118992    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.119093 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.119022    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b99\" (UniqueName: \"kubernetes.io/projected/493e6f17-b1a7-4f9c-88b0-effe1d90678f-kube-api-access-t8b99\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.119093 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.119084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.119236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.119110    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.219993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.219957    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.219993 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.219997    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220023    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220047    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220163    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220252 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220204    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b99\" (UniqueName: \"kubernetes.io/projected/493e6f17-b1a7-4f9c-88b0-effe1d90678f-kube-api-access-t8b99\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220467 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220434    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-home\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220523    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.220604 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.220562    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.222577 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.222554    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.222723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.222705    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.233242 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.233220    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b99\" (UniqueName: \"kubernetes.io/projected/493e6f17-b1a7-4f9c-88b0-effe1d90678f-kube-api-access-t8b99\") pod \"scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.349969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.349882    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:01.486400 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.486370    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"]
Jul 27 17:28:01.488548 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:28:01.488518    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493e6f17_b1a7_4f9c_88b0_effe1d90678f.slice/crio-521258b46ecc3bf4a0b1a0353e19c17d2ed67d370ba00f1677cb384d4b5ad7e4 WatchSource:0}: Error finding container 521258b46ecc3bf4a0b1a0353e19c17d2ed67d370ba00f1677cb384d4b5ad7e4: Status 404 returned error can't find the container with id 521258b46ecc3bf4a0b1a0353e19c17d2ed67d370ba00f1677cb384d4b5ad7e4
Jul 27 17:28:01.624449 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.624356    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" event={"ID":"493e6f17-b1a7-4f9c-88b0-effe1d90678f","Type":"ContainerStarted","Data":"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"}
Jul 27 17:28:01.624449 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.624411    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" event={"ID":"493e6f17-b1a7-4f9c-88b0-effe1d90678f","Type":"ContainerStarted","Data":"521258b46ecc3bf4a0b1a0353e19c17d2ed67d370ba00f1677cb384d4b5ad7e4"}
Jul 27 17:28:01.624764 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:01.624456    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:02.630643 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:02.630606    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" event={"ID":"493e6f17-b1a7-4f9c-88b0-effe1d90678f","Type":"ContainerStarted","Data":"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"}
Jul 27 17:28:02.652197 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:02.652141    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" podStartSLOduration=1.652126633 podStartE2EDuration="1.652126633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:28:02.650708053 +0000 UTC m=+1591.236069610" watchObservedRunningTime="2026-07-27 17:28:02.652126633 +0000 UTC m=+1591.237488189"
Jul 27 17:28:03.350304 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:03.350268    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:11.350642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:11.350600    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:11.369626 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:11.369597    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:11.370765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:11.370744    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:11.685208 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:11.685123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:34.831441 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:34.831401    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"]
Jul 27 17:28:34.842077 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:28:34.842022    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:28:34.842077 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:28:34.842077 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:28:34.842077 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a" cmd=["/bin/sleep","15"]
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:34.842088    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" message="" err=<
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:28:34.842101    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="main" containerID="cri-o://552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"
Jul 27 17:28:34.842419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:34.842149    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="main" containerID="cri-o://552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a" gracePeriod=60
Jul 27 17:28:35.455088 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.455038    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="llm-d-routing-sidecar" containerID="cri-o://7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71" gracePeriod=60
Jul 27 17:28:35.701773 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.701750    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:35.784173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784088    2579 generic.go:356] "Generic (PLEG): container finished" podID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerID="552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a" exitCode=0
Jul 27 17:28:35.784173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784112    2579 generic.go:356] "Generic (PLEG): container finished" podID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerID="7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71" exitCode=0
Jul 27 17:28:35.784173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784130    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"
Jul 27 17:28:35.784173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784150    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" event={"ID":"493e6f17-b1a7-4f9c-88b0-effe1d90678f","Type":"ContainerDied","Data":"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"}
Jul 27 17:28:35.784456 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784190    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" event={"ID":"493e6f17-b1a7-4f9c-88b0-effe1d90678f","Type":"ContainerDied","Data":"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"}
Jul 27 17:28:35.784456 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784208    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" event={"ID":"493e6f17-b1a7-4f9c-88b0-effe1d90678f","Type":"ContainerDied","Data":"521258b46ecc3bf4a0b1a0353e19c17d2ed67d370ba00f1677cb384d4b5ad7e4"}
Jul 27 17:28:35.784456 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.784228    2579 scope.go:117] "RemoveContainer" containerID="552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"
Jul 27 17:28:35.792384 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.792357    2579 scope.go:117] "RemoveContainer" containerID="7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"
Jul 27 17:28:35.799358 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.799342    2579 scope.go:117] "RemoveContainer" containerID="552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"
Jul 27 17:28:35.799615 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:28:35.799598    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a\": container with ID starting with 552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a not found: ID does not exist" containerID="552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"
Jul 27 17:28:35.799667 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.799623    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"} err="failed to get container status \"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a\": rpc error: code = NotFound desc = could not find container \"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a\": container with ID starting with 552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a not found: ID does not exist"
Jul 27 17:28:35.799667 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.799638    2579 scope.go:117] "RemoveContainer" containerID="7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"
Jul 27 17:28:35.799884 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:28:35.799865    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71\": container with ID starting with 7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71 not found: ID does not exist" containerID="7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"
Jul 27 17:28:35.799927 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.799890    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"} err="failed to get container status \"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71\": rpc error: code = NotFound desc = could not find container \"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71\": container with ID starting with 7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71 not found: ID does not exist"
Jul 27 17:28:35.799927 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.799905    2579 scope.go:117] "RemoveContainer" containerID="552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"
Jul 27 17:28:35.800131 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.800113    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a"} err="failed to get container status \"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a\": rpc error: code = NotFound desc = could not find container \"552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a\": container with ID starting with 552204f9aa3d69019194ace21a4365df947e6f146f3627896a7828138473d42a not found: ID does not exist"
Jul 27 17:28:35.800176 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.800132    2579 scope.go:117] "RemoveContainer" containerID="7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"
Jul 27 17:28:35.800316 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.800300    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71"} err="failed to get container status \"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71\": rpc error: code = NotFound desc = could not find container \"7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71\": container with ID starting with 7af5c624059036683af4c9d9a785af915f8932f8f8020bca42bda5218eb36d71 not found: ID does not exist"
Jul 27 17:28:35.803564 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803548    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-model-cache\") pod \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") "
Jul 27 17:28:35.803618 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803611    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8b99\" (UniqueName: \"kubernetes.io/projected/493e6f17-b1a7-4f9c-88b0-effe1d90678f-kube-api-access-t8b99\") pod \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") "
Jul 27 17:28:35.803662 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803636    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-home\") pod \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") "
Jul 27 17:28:35.803714 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803669    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-dshm\") pod \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") "
Jul 27 17:28:35.803714 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803699    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tls-certs\") pod \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") "
Jul 27 17:28:35.803819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tmp-dir\") pod \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\" (UID: \"493e6f17-b1a7-4f9c-88b0-effe1d90678f\") "
Jul 27 17:28:35.803819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803726    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-model-cache" (OuterVolumeSpecName: "model-cache") pod "493e6f17-b1a7-4f9c-88b0-effe1d90678f" (UID: "493e6f17-b1a7-4f9c-88b0-effe1d90678f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.803924 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803879    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.803924 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.803913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-home" (OuterVolumeSpecName: "home") pod "493e6f17-b1a7-4f9c-88b0-effe1d90678f" (UID: "493e6f17-b1a7-4f9c-88b0-effe1d90678f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.804024 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.804004    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "493e6f17-b1a7-4f9c-88b0-effe1d90678f" (UID: "493e6f17-b1a7-4f9c-88b0-effe1d90678f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.805781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.805761    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-dshm" (OuterVolumeSpecName: "dshm") pod "493e6f17-b1a7-4f9c-88b0-effe1d90678f" (UID: "493e6f17-b1a7-4f9c-88b0-effe1d90678f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:28:35.805864 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.805847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "493e6f17-b1a7-4f9c-88b0-effe1d90678f" (UID: "493e6f17-b1a7-4f9c-88b0-effe1d90678f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:28:35.805994 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.805979    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493e6f17-b1a7-4f9c-88b0-effe1d90678f-kube-api-access-t8b99" (OuterVolumeSpecName: "kube-api-access-t8b99") pod "493e6f17-b1a7-4f9c-88b0-effe1d90678f" (UID: "493e6f17-b1a7-4f9c-88b0-effe1d90678f"). InnerVolumeSpecName "kube-api-access-t8b99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:28:35.905229 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.905204    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8b99\" (UniqueName: \"kubernetes.io/projected/493e6f17-b1a7-4f9c-88b0-effe1d90678f-kube-api-access-t8b99\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.905229 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.905228    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.905597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.905238    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.905597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.905246    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:28:35.905597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:35.905255    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/493e6f17-b1a7-4f9c-88b0-effe1d90678f-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:28:36.100966 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:36.100884    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"]
Jul 27 17:28:36.105256 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:36.105227    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc"]
Jul 27 17:28:36.675003 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:36.674947    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-64497f9569-4k7gc" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8001/health\": context deadline exceeded"
Jul 27 17:28:37.893154 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:28:37.893117    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" path="/var/lib/kubelet/pods/493e6f17-b1a7-4f9c-88b0-effe1d90678f/volumes"
Jul 27 17:29:06.636463 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636427    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"]
Jul 27 17:29:06.636963 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636749    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="llm-d-routing-sidecar"
Jul 27 17:29:06.636963 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636759    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="llm-d-routing-sidecar"
Jul 27 17:29:06.636963 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636773    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="main"
Jul 27 17:29:06.636963 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636778    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="main"
Jul 27 17:29:06.636963 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636858    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="main"
Jul 27 17:29:06.636963 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.636867    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="493e6f17-b1a7-4f9c-88b0-effe1d90678f" containerName="llm-d-routing-sidecar"
Jul 27 17:29:06.639815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.639798    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.649996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.649974    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"seaweedfs-s3-creds\""
Jul 27 17:29:06.650096 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.650015    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Jul 27 17:29:06.651771 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.651751    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Jul 27 17:29:06.651870 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.651829    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Jul 27 17:29:06.659245 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.659230    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-945qz\""
Jul 27 17:29:06.660525 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.660574 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660553    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.660621 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660572    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4wj2\" (UniqueName: \"kubernetes.io/projected/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kube-api-access-p4wj2\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.660621 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660590    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-home\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.660621 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-model-cache\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.660729 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.660774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.660749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-dshm\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.693900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.693874    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"]
Jul 27 17:29:06.761585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761550    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-dshm\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.761781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761595    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.761781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761634    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.761781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761660    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4wj2\" (UniqueName: \"kubernetes.io/projected/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kube-api-access-p4wj2\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.761781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761684    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-home\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.761781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761717    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-model-cache\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.761781 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.761743    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.762116 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.762092    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kserve-provision-location\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.762205 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.762189    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-home\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.762257 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.762195    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-model-cache\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.762295 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.762251    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tmp-dir\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.764004 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.763987    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-dshm\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.764385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.764362    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tls-certs\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.784052 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.784019    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4wj2\" (UniqueName: \"kubernetes.io/projected/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kube-api-access-p4wj2\") pod \"tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:06.949815 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:06.949720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:07.036057 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.036021    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"]
Jul 27 17:29:07.041399 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.039660    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.044113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.044049    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-gpdm2\""
Jul 27 17:29:07.051224 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.050902    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"]
Jul 27 17:29:07.064348 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.064322    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad41c77-1a75-4ad3-bb39-ea4270605b54-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.064632 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.064610    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6nls\" (UniqueName: \"kubernetes.io/projected/cad41c77-1a75-4ad3-bb39-ea4270605b54-kube-api-access-q6nls\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.097030 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.097010    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"]
Jul 27 17:29:07.099037 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:29:07.099011    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5301fa6_6d2a_4d7a_89c6_a7bdde7ddb07.slice/crio-412507b13757f84d656bf60bcf7b1f0e47631d2235238ea4e92c01cd7affcb23 WatchSource:0}: Error finding container 412507b13757f84d656bf60bcf7b1f0e47631d2235238ea4e92c01cd7affcb23: Status 404 returned error can't find the container with id 412507b13757f84d656bf60bcf7b1f0e47631d2235238ea4e92c01cd7affcb23
Jul 27 17:29:07.165791 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.165763    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad41c77-1a75-4ad3-bb39-ea4270605b54-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.165918 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.165802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6nls\" (UniqueName: \"kubernetes.io/projected/cad41c77-1a75-4ad3-bb39-ea4270605b54-kube-api-access-q6nls\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.168251 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.168228    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad41c77-1a75-4ad3-bb39-ea4270605b54-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.173991 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.173973    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6nls\" (UniqueName: \"kubernetes.io/projected/cad41c77-1a75-4ad3-bb39-ea4270605b54-kube-api-access-q6nls\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.353059 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.353019    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.500983 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.500928    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"]
Jul 27 17:29:07.505169 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:29:07.505138    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad41c77_1a75_4ad3_bb39_ea4270605b54.slice/crio-be024b07f587db33e9f82d89e24d5878365156347ea082ccc0d593390aef9040 WatchSource:0}: Error finding container be024b07f587db33e9f82d89e24d5878365156347ea082ccc0d593390aef9040: Status 404 returned error can't find the container with id be024b07f587db33e9f82d89e24d5878365156347ea082ccc0d593390aef9040
Jul 27 17:29:07.924550 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.924517    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" event={"ID":"cad41c77-1a75-4ad3-bb39-ea4270605b54","Type":"ContainerStarted","Data":"1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95"}
Jul 27 17:29:07.924988 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.924574    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" event={"ID":"cad41c77-1a75-4ad3-bb39-ea4270605b54","Type":"ContainerStarted","Data":"be024b07f587db33e9f82d89e24d5878365156347ea082ccc0d593390aef9040"}
Jul 27 17:29:07.924988 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.924612    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:29:07.926037 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.926015    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" event={"ID":"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07","Type":"ContainerStarted","Data":"e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f"}
Jul 27 17:29:07.926155 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.926052    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" event={"ID":"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07","Type":"ContainerStarted","Data":"412507b13757f84d656bf60bcf7b1f0e47631d2235238ea4e92c01cd7affcb23"}
Jul 27 17:29:07.944590 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:07.944525    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" podStartSLOduration=0.944507302 podStartE2EDuration="944.507302ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:07.941465169 +0000 UTC m=+1656.526826740" watchObservedRunningTime="2026-07-27 17:29:07.944507302 +0000 UTC m=+1656.529868861"
Jul 27 17:29:12.950307 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:12.950200    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerID="e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f" exitCode=0
Jul 27 17:29:12.950307 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:12.950280    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" event={"ID":"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07","Type":"ContainerDied","Data":"e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f"}
Jul 27 17:29:13.956518 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:13.956458    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" event={"ID":"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07","Type":"ContainerStarted","Data":"6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"}
Jul 27 17:29:13.978694 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:13.978631    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" podStartSLOduration=2.617917116 podStartE2EDuration="7.978613739s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.360696623s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:29:13.977854966 +0000 UTC m=+1662.563216514" watchObservedRunningTime="2026-07-27 17:29:13.978613739 +0000 UTC m=+1662.563975295"
Jul 27 17:29:14.950819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:14.950777    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:16.950253 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:16.950204    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:16.962813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:16.962788    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:16.987710 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:16.987683    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:29:38.932232 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:29:38.932200    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:30:28.565780 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:28.565731    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"]
Jul 27 17:30:28.567654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:28.567629    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"]
Jul 27 17:30:28.578939 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:30:28.578897    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:30:28.578939 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:30:28.578939 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:30:28.578939 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1" cmd=["/bin/sleep","15"]
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:28.578965    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" message="" err=<
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:30:28.578981    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="main" containerID="cri-o://6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"
Jul 27 17:30:28.579236 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:28.579039    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="main" containerID="cri-o://6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1" gracePeriod=60
Jul 27 17:30:28.931774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:28.931740    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:30:28.938964 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:28.938942    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:30:29.083437 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083401    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-model-cache\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083450    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4wj2\" (UniqueName: \"kubernetes.io/projected/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kube-api-access-p4wj2\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083468    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-dshm\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083499    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-home\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tls-certs\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083575    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kserve-provision-location\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083656 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083629    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tmp-dir\") pod \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\" (UID: \"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07\") "
Jul 27 17:30:29.083972 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083780    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.083972 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083793    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-home" (OuterVolumeSpecName: "home") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.083972 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083966    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.084098 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.083984    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.084098 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.084011    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.085941 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.085913    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kube-api-access-p4wj2" (OuterVolumeSpecName: "kube-api-access-p4wj2") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "kube-api-access-p4wj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:29.086154 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.086129    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:29.086154 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.086139    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-dshm" (OuterVolumeSpecName: "dshm") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.149607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.149510    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" (UID: "c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:30:29.184689 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.184654    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4wj2\" (UniqueName: \"kubernetes.io/projected/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kube-api-access-p4wj2\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.184689 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.184688    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.184897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.184702    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.184897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.184715    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-kserve-provision-location\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.184897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.184748    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:29.293745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.293702    2579 generic.go:356] "Generic (PLEG): container finished" podID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerID="6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1" exitCode=0
Jul 27 17:30:29.293946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.293781    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"
Jul 27 17:30:29.293946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.293824    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" event={"ID":"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07","Type":"ContainerDied","Data":"6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"}
Jul 27 17:30:29.293946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.293903    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl" event={"ID":"c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07","Type":"ContainerDied","Data":"412507b13757f84d656bf60bcf7b1f0e47631d2235238ea4e92c01cd7affcb23"}
Jul 27 17:30:29.293946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.293926    2579 scope.go:117] "RemoveContainer" containerID="6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"
Jul 27 17:30:29.302951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.302926    2579 scope.go:117] "RemoveContainer" containerID="e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f"
Jul 27 17:30:29.310702 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.310684    2579 scope.go:117] "RemoveContainer" containerID="6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"
Jul 27 17:30:29.310955 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:30:29.310938    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1\": container with ID starting with 6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1 not found: ID does not exist" containerID="6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"
Jul 27 17:30:29.311001 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.310962    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1"} err="failed to get container status \"6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1\": rpc error: code = NotFound desc = could not find container \"6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1\": container with ID starting with 6285640dfbedcfa09c35795415c4fcdbdb6e13afd02b382166a5e5f6a5cab3c1 not found: ID does not exist"
Jul 27 17:30:29.311001 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.310980    2579 scope.go:117] "RemoveContainer" containerID="e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f"
Jul 27 17:30:29.311204 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:30:29.311186    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f\": container with ID starting with e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f not found: ID does not exist" containerID="e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f"
Jul 27 17:30:29.311248 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.311212    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f"} err="failed to get container status \"e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f\": rpc error: code = NotFound desc = could not find container \"e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f\": container with ID starting with e7bda1d3702e9f6a6863fb8079e1bcb6af1aee0d06db1eb27eec4ea59850253f not found: ID does not exist"
Jul 27 17:30:29.316947 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.316926    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"]
Jul 27 17:30:29.326714 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.326689    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-74757d9fd8-2cqhl"]
Jul 27 17:30:29.893202 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:29.893174    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" path="/var/lib/kubelet/pods/c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07/volumes"
Jul 27 17:30:38.931616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:38.931580    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:30:43.588133 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.588092    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerName="main" containerID="cri-o://1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95" gracePeriod=45
Jul 27 17:30:43.840763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.840700    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:30:43.895676 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.895651    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad41c77-1a75-4ad3-bb39-ea4270605b54-tls-certs\") pod \"cad41c77-1a75-4ad3-bb39-ea4270605b54\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") "
Jul 27 17:30:43.895849 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.895682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6nls\" (UniqueName: \"kubernetes.io/projected/cad41c77-1a75-4ad3-bb39-ea4270605b54-kube-api-access-q6nls\") pod \"cad41c77-1a75-4ad3-bb39-ea4270605b54\" (UID: \"cad41c77-1a75-4ad3-bb39-ea4270605b54\") "
Jul 27 17:30:43.897881 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.897847    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cad41c77-1a75-4ad3-bb39-ea4270605b54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cad41c77-1a75-4ad3-bb39-ea4270605b54" (UID: "cad41c77-1a75-4ad3-bb39-ea4270605b54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:30:43.897989 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.897905    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cad41c77-1a75-4ad3-bb39-ea4270605b54-kube-api-access-q6nls" (OuterVolumeSpecName: "kube-api-access-q6nls") pod "cad41c77-1a75-4ad3-bb39-ea4270605b54" (UID: "cad41c77-1a75-4ad3-bb39-ea4270605b54"). InnerVolumeSpecName "kube-api-access-q6nls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:30:43.996419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.996381    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cad41c77-1a75-4ad3-bb39-ea4270605b54-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:43.996419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:43.996411    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6nls\" (UniqueName: \"kubernetes.io/projected/cad41c77-1a75-4ad3-bb39-ea4270605b54-kube-api-access-q6nls\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:30:44.361139 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.361094    2579 generic.go:356] "Generic (PLEG): container finished" podID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerID="1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95" exitCode=0
Jul 27 17:30:44.361341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.361157    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"
Jul 27 17:30:44.361341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.361158    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" event={"ID":"cad41c77-1a75-4ad3-bb39-ea4270605b54","Type":"ContainerDied","Data":"1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95"}
Jul 27 17:30:44.361341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.361229    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8" event={"ID":"cad41c77-1a75-4ad3-bb39-ea4270605b54","Type":"ContainerDied","Data":"be024b07f587db33e9f82d89e24d5878365156347ea082ccc0d593390aef9040"}
Jul 27 17:30:44.361341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.361260    2579 scope.go:117] "RemoveContainer" containerID="1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95"
Jul 27 17:30:44.370370 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.370352    2579 scope.go:117] "RemoveContainer" containerID="1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95"
Jul 27 17:30:44.370657 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:30:44.370639    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95\": container with ID starting with 1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95 not found: ID does not exist" containerID="1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95"
Jul 27 17:30:44.370727 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.370667    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95"} err="failed to get container status \"1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95\": rpc error: code = NotFound desc = could not find container \"1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95\": container with ID starting with 1a628ad5b9192d0ed8183de7181f936761462f8a3d1f979bc3a697ef73ddcb95 not found: ID does not exist"
Jul 27 17:30:44.383506 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.383461    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"]
Jul 27 17:30:44.387666 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:44.387648    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-869cbf5fq4cn8"]
Jul 27 17:30:45.892556 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:30:45.892523    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" path="/var/lib/kubelet/pods/cad41c77-1a75-4ad3-bb39-ea4270605b54/volumes"
Jul 27 17:31:07.725772 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.725738    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"]
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726040    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="storage-initializer"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726051    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="storage-initializer"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726065    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerName="main"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726071    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerName="main"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726081    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="main"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726086    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="main"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726135    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5301fa6-6d2a-4d7a-89c6-a7bdde7ddb07" containerName="main"
Jul 27 17:31:07.726142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.726144    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="cad41c77-1a75-4ad3-bb39-ea4270605b54" containerName="main"
Jul 27 17:31:07.733419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.733392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.736331 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.736310    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-4tfpm\""
Jul 27 17:31:07.736464 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.736368    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"seaweedfs-s3-creds\""
Jul 27 17:31:07.737385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.737364    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Jul 27 17:31:07.737385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.737376    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Jul 27 17:31:07.737515 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.737370    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Jul 27 17:31:07.744245 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.744222    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"]
Jul 27 17:31:07.796869 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.796836    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cjfw\" (UniqueName: \"kubernetes.io/projected/c44cc845-2d54-43df-a243-85bd5c52aa15-kube-api-access-2cjfw\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.797038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.796884    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cc845-2d54-43df-a243-85bd5c52aa15-tls-certs\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.797038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.796914    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-model-cache\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.797038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.796967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.797038 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.797021    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-dshm\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.797247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.797077    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-home\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.797247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.797106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898507    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-model-cache\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898537    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898562    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-dshm\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898598    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-home\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898624    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898678    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cjfw\" (UniqueName: \"kubernetes.io/projected/c44cc845-2d54-43df-a243-85bd5c52aa15-kube-api-access-2cjfw\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.898774 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cc845-2d54-43df-a243-85bd5c52aa15-tls-certs\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.899022 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.898938    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-model-cache\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.899022 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.899006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-tmp-dir\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.899106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.899051    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-home\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.899142 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.899109    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-kserve-provision-location\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.901017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.900995    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-dshm\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.901363 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.901342    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cc845-2d54-43df-a243-85bd5c52aa15-tls-certs\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.906888 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.906869    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cjfw\" (UniqueName: \"kubernetes.io/projected/c44cc845-2d54-43df-a243-85bd5c52aa15-kube-api-access-2cjfw\") pod \"tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:07.969286 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.969248    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"]
Jul 27 17:31:07.973083 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.973061    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:07.975959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.975897    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-ljxzk\""
Jul 27 17:31:07.982158 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:07.982134    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"]
Jul 27 17:31:08.044860 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.044831    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:08.100076 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.100045    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f3dae1-632d-43a8-a006-3c69674d2b0a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.100223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.100100    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzz8f\" (UniqueName: \"kubernetes.io/projected/61f3dae1-632d-43a8-a006-3c69674d2b0a-kube-api-access-hzz8f\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.178147 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.178125    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"]
Jul 27 17:31:08.180048 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:31:08.180020    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44cc845_2d54_43df_a243_85bd5c52aa15.slice/crio-1a7ad2d3494a4149de0fa91e04c6c92576fc98cd2b983e6a991254a2ccd76bad WatchSource:0}: Error finding container 1a7ad2d3494a4149de0fa91e04c6c92576fc98cd2b983e6a991254a2ccd76bad: Status 404 returned error can't find the container with id 1a7ad2d3494a4149de0fa91e04c6c92576fc98cd2b983e6a991254a2ccd76bad
Jul 27 17:31:08.201343 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.201321    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f3dae1-632d-43a8-a006-3c69674d2b0a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.201450 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.201350    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzz8f\" (UniqueName: \"kubernetes.io/projected/61f3dae1-632d-43a8-a006-3c69674d2b0a-kube-api-access-hzz8f\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.203825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.203800    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f3dae1-632d-43a8-a006-3c69674d2b0a-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.211444 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.211423    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzz8f\" (UniqueName: \"kubernetes.io/projected/61f3dae1-632d-43a8-a006-3c69674d2b0a-kube-api-access-hzz8f\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.284642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.284615    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:08.422186 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.422162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"]
Jul 27 17:31:08.424604 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:31:08.424576    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f3dae1_632d_43a8_a006_3c69674d2b0a.slice/crio-21697877ca862518b1106a9bada4d4227987cb661e251363ae04807b848937d7 WatchSource:0}: Error finding container 21697877ca862518b1106a9bada4d4227987cb661e251363ae04807b848937d7: Status 404 returned error can't find the container with id 21697877ca862518b1106a9bada4d4227987cb661e251363ae04807b848937d7
Jul 27 17:31:08.466274 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.466244    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" event={"ID":"61f3dae1-632d-43a8-a006-3c69674d2b0a","Type":"ContainerStarted","Data":"21697877ca862518b1106a9bada4d4227987cb661e251363ae04807b848937d7"}
Jul 27 17:31:08.467859 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.467831    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" event={"ID":"c44cc845-2d54-43df-a243-85bd5c52aa15","Type":"ContainerStarted","Data":"1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121"}
Jul 27 17:31:08.467945 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:08.467891    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" event={"ID":"c44cc845-2d54-43df-a243-85bd5c52aa15","Type":"ContainerStarted","Data":"1a7ad2d3494a4149de0fa91e04c6c92576fc98cd2b983e6a991254a2ccd76bad"}
Jul 27 17:31:09.475677 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:09.475643    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" event={"ID":"61f3dae1-632d-43a8-a006-3c69674d2b0a","Type":"ContainerStarted","Data":"cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0"}
Jul 27 17:31:09.476181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:09.475930    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:09.516398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:09.516342    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" podStartSLOduration=2.51632108 podStartE2EDuration="2.51632108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:09.515160599 +0000 UTC m=+1778.100522157" watchObservedRunningTime="2026-07-27 17:31:09.51632108 +0000 UTC m=+1778.101682637"
Jul 27 17:31:16.509569 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:16.509534    2579 generic.go:356] "Generic (PLEG): container finished" podID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerID="1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121" exitCode=0
Jul 27 17:31:16.509998 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:16.509579    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" event={"ID":"c44cc845-2d54-43df-a243-85bd5c52aa15","Type":"ContainerDied","Data":"1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121"}
Jul 27 17:31:17.522124 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:17.522087    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" event={"ID":"c44cc845-2d54-43df-a243-85bd5c52aa15","Type":"ContainerStarted","Data":"551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"}
Jul 27 17:31:17.542422 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:17.542366    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" podStartSLOduration=2.6410484309999998 podStartE2EDuration="10.54234992s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.901301489s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:31:17.541527543 +0000 UTC m=+1786.126889100" watchObservedRunningTime="2026-07-27 17:31:17.54234992 +0000 UTC m=+1786.127711506"
Jul 27 17:31:18.045180 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:18.045147    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:18.045358 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:18.045210    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:18.059218 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:18.059193    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:18.538162 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:18.538133    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:31.861667 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:31.861638    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:31:31.868921 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:31.868898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:31:40.487527 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:40.483869    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:31:51.038575 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.038540    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"]
Jul 27 17:31:51.041159 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.041130    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"]
Jul 27 17:31:51.051486 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:31:51.051431    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:31:51.051486 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:31:51.051486 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:31:51.051486 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385" cmd=["/bin/sleep","15"]
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.051586    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" message="" err=<
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:31:51.051604    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="main" containerID="cri-o://551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"
Jul 27 17:31:51.051726 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.051647    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="main" containerID="cri-o://551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385" gracePeriod=60
Jul 27 17:31:51.406646 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.406624    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:51.488948 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.488919    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-kserve-provision-location\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.488970    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cjfw\" (UniqueName: \"kubernetes.io/projected/c44cc845-2d54-43df-a243-85bd5c52aa15-kube-api-access-2cjfw\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489025    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-home\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489052    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-dshm\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489089    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cc845-2d54-43df-a243-85bd5c52aa15-tls-certs\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489151 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489144    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-model-cache\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489423 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489168    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-tmp-dir\") pod \"c44cc845-2d54-43df-a243-85bd5c52aa15\" (UID: \"c44cc845-2d54-43df-a243-85bd5c52aa15\") "
Jul 27 17:31:51.489423 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489331    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-home" (OuterVolumeSpecName: "home") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.489563 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489528    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.489616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489551    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-model-cache" (OuterVolumeSpecName: "model-cache") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.489616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.489542    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.491668 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.491630    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44cc845-2d54-43df-a243-85bd5c52aa15-kube-api-access-2cjfw" (OuterVolumeSpecName: "kube-api-access-2cjfw") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "kube-api-access-2cjfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:31:51.491784 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.491650    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-dshm" (OuterVolumeSpecName: "dshm") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.491784 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.491714    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44cc845-2d54-43df-a243-85bd5c52aa15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:31:51.547334 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.547298    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c44cc845-2d54-43df-a243-85bd5c52aa15" (UID: "c44cc845-2d54-43df-a243-85bd5c52aa15"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:31:51.590970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.590881    2579 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-kserve-provision-location\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.590970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.590921    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cjfw\" (UniqueName: \"kubernetes.io/projected/c44cc845-2d54-43df-a243-85bd5c52aa15-kube-api-access-2cjfw\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.590970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.590934    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.590970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.590947    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c44cc845-2d54-43df-a243-85bd5c52aa15-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.590970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.590960    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.590970 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.590971    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c44cc845-2d54-43df-a243-85bd5c52aa15-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:31:51.673396 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.673361    2579 generic.go:356] "Generic (PLEG): container finished" podID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerID="551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385" exitCode=0
Jul 27 17:31:51.673586 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.673419    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"
Jul 27 17:31:51.673586 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.673490    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" event={"ID":"c44cc845-2d54-43df-a243-85bd5c52aa15","Type":"ContainerDied","Data":"551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"}
Jul 27 17:31:51.673586 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.673554    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7" event={"ID":"c44cc845-2d54-43df-a243-85bd5c52aa15","Type":"ContainerDied","Data":"1a7ad2d3494a4149de0fa91e04c6c92576fc98cd2b983e6a991254a2ccd76bad"}
Jul 27 17:31:51.673586 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.673576    2579 scope.go:117] "RemoveContainer" containerID="551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"
Jul 27 17:31:51.683351 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.683331    2579 scope.go:117] "RemoveContainer" containerID="1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121"
Jul 27 17:31:51.691611 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.691592    2579 scope.go:117] "RemoveContainer" containerID="551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"
Jul 27 17:31:51.691902 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:31:51.691882    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385\": container with ID starting with 551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385 not found: ID does not exist" containerID="551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"
Jul 27 17:31:51.691988 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.691914    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385"} err="failed to get container status \"551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385\": rpc error: code = NotFound desc = could not find container \"551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385\": container with ID starting with 551f4097f372d4d11762b58c85a5409829c9ffa002b6d09a0a9b8944aab26385 not found: ID does not exist"
Jul 27 17:31:51.691988 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.691940    2579 scope.go:117] "RemoveContainer" containerID="1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121"
Jul 27 17:31:51.692219 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:31:51.692201    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121\": container with ID starting with 1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121 not found: ID does not exist" containerID="1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121"
Jul 27 17:31:51.692273 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.692229    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121"} err="failed to get container status \"1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121\": rpc error: code = NotFound desc = could not find container \"1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121\": container with ID starting with 1a68695e6e639f181e58df10006d5d3c6c45ba8121fa6e5eeed8d344b6412121 not found: ID does not exist"
Jul 27 17:31:51.698635 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.698613    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"]
Jul 27 17:31:51.704764 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.704739    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-554bbc7c45-gkrh7"]
Jul 27 17:31:51.893752 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:31:51.893646    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" path="/var/lib/kubelet/pods/c44cc845-2d54-43df-a243-85bd5c52aa15/volumes"
Jul 27 17:32:00.482639 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:00.482597    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" podUID="61f3dae1-632d-43a8-a006-3c69674d2b0a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:32:06.059659 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.059588    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" podUID="61f3dae1-632d-43a8-a006-3c69674d2b0a" containerName="main" containerID="cri-o://cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0" gracePeriod=45
Jul 27 17:32:06.329009 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.328982    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:32:06.409681 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.409647    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzz8f\" (UniqueName: \"kubernetes.io/projected/61f3dae1-632d-43a8-a006-3c69674d2b0a-kube-api-access-hzz8f\") pod \"61f3dae1-632d-43a8-a006-3c69674d2b0a\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") "
Jul 27 17:32:06.409870 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.409750    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f3dae1-632d-43a8-a006-3c69674d2b0a-tls-certs\") pod \"61f3dae1-632d-43a8-a006-3c69674d2b0a\" (UID: \"61f3dae1-632d-43a8-a006-3c69674d2b0a\") "
Jul 27 17:32:06.411978 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.411943    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f3dae1-632d-43a8-a006-3c69674d2b0a-kube-api-access-hzz8f" (OuterVolumeSpecName: "kube-api-access-hzz8f") pod "61f3dae1-632d-43a8-a006-3c69674d2b0a" (UID: "61f3dae1-632d-43a8-a006-3c69674d2b0a"). InnerVolumeSpecName "kube-api-access-hzz8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:32:06.412086 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.411993    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f3dae1-632d-43a8-a006-3c69674d2b0a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61f3dae1-632d-43a8-a006-3c69674d2b0a" (UID: "61f3dae1-632d-43a8-a006-3c69674d2b0a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:32:06.510523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.510463    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzz8f\" (UniqueName: \"kubernetes.io/projected/61f3dae1-632d-43a8-a006-3c69674d2b0a-kube-api-access-hzz8f\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:32:06.510523 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.510518    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f3dae1-632d-43a8-a006-3c69674d2b0a-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:32:06.739433 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.739394    2579 generic.go:356] "Generic (PLEG): container finished" podID="61f3dae1-632d-43a8-a006-3c69674d2b0a" containerID="cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0" exitCode=0
Jul 27 17:32:06.739433 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.739433    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" event={"ID":"61f3dae1-632d-43a8-a006-3c69674d2b0a","Type":"ContainerDied","Data":"cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0"}
Jul 27 17:32:06.739719 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.739448    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"
Jul 27 17:32:06.739719 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.739498    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f" event={"ID":"61f3dae1-632d-43a8-a006-3c69674d2b0a","Type":"ContainerDied","Data":"21697877ca862518b1106a9bada4d4227987cb661e251363ae04807b848937d7"}
Jul 27 17:32:06.739719 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.739525    2579 scope.go:117] "RemoveContainer" containerID="cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0"
Jul 27 17:32:06.748424 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.748404    2579 scope.go:117] "RemoveContainer" containerID="cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0"
Jul 27 17:32:06.748747 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:32:06.748726    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0\": container with ID starting with cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0 not found: ID does not exist" containerID="cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0"
Jul 27 17:32:06.748813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.748759    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0"} err="failed to get container status \"cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0\": rpc error: code = NotFound desc = could not find container \"cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0\": container with ID starting with cf556290710fe01127739325736753a213f430f8cefc2c4fee340fb189bd89a0 not found: ID does not exist"
Jul 27 17:32:06.760285 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.760250    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"]
Jul 27 17:32:06.764402 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:06.764381    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-56bb2xk2f"]
Jul 27 17:32:07.892113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:07.892078    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f3dae1-632d-43a8-a006-3c69674d2b0a" path="/var/lib/kubelet/pods/61f3dae1-632d-43a8-a006-3c69674d2b0a/volumes"
Jul 27 17:32:22.492138 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492107    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"]
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492417    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="storage-initializer"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492428    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="storage-initializer"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492448    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f3dae1-632d-43a8-a006-3c69674d2b0a" containerName="main"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492455    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f3dae1-632d-43a8-a006-3c69674d2b0a" containerName="main"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492463    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="main"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492486    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="main"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492546    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f3dae1-632d-43a8-a006-3c69674d2b0a" containerName="main"
Jul 27 17:32:22.492637 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.492556    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44cc845-2d54-43df-a243-85bd5c52aa15" containerName="main"
Jul 27 17:32:22.496413 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.496392    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.500579 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.500557    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Jul 27 17:32:22.500708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.500583    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Jul 27 17:32:22.500708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.500558    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Jul 27 17:32:22.500708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.500558    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-lpgtt\""
Jul 27 17:32:22.500708 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.500557    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-qss6b\""
Jul 27 17:32:22.503939 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.503922    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"]
Jul 27 17:32:22.640711 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.640682    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqsw4\" (UniqueName: \"kubernetes.io/projected/d75c691c-9d07-4619-b1c8-9428041d059c-kube-api-access-nqsw4\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.640865 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.640724    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d75c691c-9d07-4619-b1c8-9428041d059c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.742237 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.742160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqsw4\" (UniqueName: \"kubernetes.io/projected/d75c691c-9d07-4619-b1c8-9428041d059c-kube-api-access-nqsw4\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.742237 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.742201    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d75c691c-9d07-4619-b1c8-9428041d059c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.744803 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.744767    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d75c691c-9d07-4619-b1c8-9428041d059c-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.750546 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.750519    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqsw4\" (UniqueName: \"kubernetes.io/projected/d75c691c-9d07-4619-b1c8-9428041d059c-kube-api-access-nqsw4\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.807056 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.807025    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:22.935358 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:22.935332    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"]
Jul 27 17:32:22.936812 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:32:22.936788    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75c691c_9d07_4619_b1c8_9428041d059c.slice/crio-444c2dae90c6eac3a704ba8b084187528ba8c80b13043565a382f51c8825b296 WatchSource:0}: Error finding container 444c2dae90c6eac3a704ba8b084187528ba8c80b13043565a382f51c8825b296: Status 404 returned error can't find the container with id 444c2dae90c6eac3a704ba8b084187528ba8c80b13043565a382f51c8825b296
Jul 27 17:32:23.816491 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:23.816429    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" event={"ID":"d75c691c-9d07-4619-b1c8-9428041d059c","Type":"ContainerStarted","Data":"db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734"}
Jul 27 17:32:23.816969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:23.816550    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" event={"ID":"d75c691c-9d07-4619-b1c8-9428041d059c","Type":"ContainerStarted","Data":"444c2dae90c6eac3a704ba8b084187528ba8c80b13043565a382f51c8825b296"}
Jul 27 17:32:23.816969 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:23.816595    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:23.832861 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:23.832806    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" podStartSLOduration=1.832790246 podStartE2EDuration="1.832790246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:32:23.832076721 +0000 UTC m=+1852.417438280" watchObservedRunningTime="2026-07-27 17:32:23.832790246 +0000 UTC m=+1852.418151801"
Jul 27 17:32:54.822467 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:54.822394    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:32:56.224275 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:32:56.224233    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"]
Jul 27 17:33:04.822457 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:04.822418    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" podUID="d75c691c-9d07-4619-b1c8-9428041d059c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:33:11.245034 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.244969    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" podUID="d75c691c-9d07-4619-b1c8-9428041d059c" containerName="main" containerID="cri-o://db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734" gracePeriod=45
Jul 27 17:33:11.495185 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.495122    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:33:11.675838 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.675785    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqsw4\" (UniqueName: \"kubernetes.io/projected/d75c691c-9d07-4619-b1c8-9428041d059c-kube-api-access-nqsw4\") pod \"d75c691c-9d07-4619-b1c8-9428041d059c\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") "
Jul 27 17:33:11.676047 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.675912    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d75c691c-9d07-4619-b1c8-9428041d059c-tls-certs\") pod \"d75c691c-9d07-4619-b1c8-9428041d059c\" (UID: \"d75c691c-9d07-4619-b1c8-9428041d059c\") "
Jul 27 17:33:11.678281 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.678249    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d75c691c-9d07-4619-b1c8-9428041d059c-kube-api-access-nqsw4" (OuterVolumeSpecName: "kube-api-access-nqsw4") pod "d75c691c-9d07-4619-b1c8-9428041d059c" (UID: "d75c691c-9d07-4619-b1c8-9428041d059c"). InnerVolumeSpecName "kube-api-access-nqsw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:33:11.678281 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.678247    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d75c691c-9d07-4619-b1c8-9428041d059c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d75c691c-9d07-4619-b1c8-9428041d059c" (UID: "d75c691c-9d07-4619-b1c8-9428041d059c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:33:11.777341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.777251    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqsw4\" (UniqueName: \"kubernetes.io/projected/d75c691c-9d07-4619-b1c8-9428041d059c-kube-api-access-nqsw4\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:33:11.777341 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:11.777284    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d75c691c-9d07-4619-b1c8-9428041d059c-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:33:12.024379 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.024339    2579 generic.go:356] "Generic (PLEG): container finished" podID="d75c691c-9d07-4619-b1c8-9428041d059c" containerID="db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734" exitCode=0
Jul 27 17:33:12.024379 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.024382    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"
Jul 27 17:33:12.024636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.024375    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" event={"ID":"d75c691c-9d07-4619-b1c8-9428041d059c","Type":"ContainerDied","Data":"db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734"}
Jul 27 17:33:12.024636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.024519    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg" event={"ID":"d75c691c-9d07-4619-b1c8-9428041d059c","Type":"ContainerDied","Data":"444c2dae90c6eac3a704ba8b084187528ba8c80b13043565a382f51c8825b296"}
Jul 27 17:33:12.024636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.024545    2579 scope.go:117] "RemoveContainer" containerID="db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734"
Jul 27 17:33:12.033207 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.033188    2579 scope.go:117] "RemoveContainer" containerID="db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734"
Jul 27 17:33:12.033484 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:33:12.033442    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734\": container with ID starting with db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734 not found: ID does not exist" containerID="db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734"
Jul 27 17:33:12.033594 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.033542    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734"} err="failed to get container status \"db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734\": rpc error: code = NotFound desc = could not find container \"db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734\": container with ID starting with db5e6db07bfdab4582a86d3a6fbd8fe7a2e6a43f3b3fd6d181fa9876d617a734 not found: ID does not exist"
Jul 27 17:33:12.041352 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.041328    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"]
Jul 27 17:33:12.044502 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:12.044467    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schel57xg"]
Jul 27 17:33:13.893536 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:13.893501    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d75c691c-9d07-4619-b1c8-9428041d059c" path="/var/lib/kubelet/pods/d75c691c-9d07-4619-b1c8-9428041d059c/volumes"
Jul 27 17:33:23.391145 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.391111    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"]
Jul 27 17:33:23.391641 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.391621    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d75c691c-9d07-4619-b1c8-9428041d059c" containerName="main"
Jul 27 17:33:23.391751 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.391644    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d75c691c-9d07-4619-b1c8-9428041d059c" containerName="main"
Jul 27 17:33:23.391751 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.391730    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d75c691c-9d07-4619-b1c8-9428041d059c" containerName="main"
Jul 27 17:33:23.396051 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.396030    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.398908 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.398885    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Jul 27 17:33:23.399660 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.399642    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-zqtkh\""
Jul 27 17:33:23.399660 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.399655    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Jul 27 17:33:23.399808 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.399686    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Jul 27 17:33:23.399808 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.399721    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-9lqrd\""
Jul 27 17:33:23.403539 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.403519    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"]
Jul 27 17:33:23.481628 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.481589    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41af1395-97b8-4816-8f63-5e2e8e078611-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.481807 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.481702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5rr\" (UniqueName: \"kubernetes.io/projected/41af1395-97b8-4816-8f63-5e2e8e078611-kube-api-access-gk5rr\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.583091 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.583049    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41af1395-97b8-4816-8f63-5e2e8e078611-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.583287 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.583130    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5rr\" (UniqueName: \"kubernetes.io/projected/41af1395-97b8-4816-8f63-5e2e8e078611-kube-api-access-gk5rr\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.585823 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.585797    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41af1395-97b8-4816-8f63-5e2e8e078611-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.594940 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.594907    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5rr\" (UniqueName: \"kubernetes.io/projected/41af1395-97b8-4816-8f63-5e2e8e078611-kube-api-access-gk5rr\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.707300 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.707265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:23.844758 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:23.844730    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"]
Jul 27 17:33:23.847081 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:33:23.847051    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41af1395_97b8_4816_8f63_5e2e8e078611.slice/crio-262bd51b5820d67d40931d34291b60b1907555acedba6c6bea1a6666fb950262 WatchSource:0}: Error finding container 262bd51b5820d67d40931d34291b60b1907555acedba6c6bea1a6666fb950262: Status 404 returned error can't find the container with id 262bd51b5820d67d40931d34291b60b1907555acedba6c6bea1a6666fb950262
Jul 27 17:33:24.082139 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:24.082044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" event={"ID":"41af1395-97b8-4816-8f63-5e2e8e078611","Type":"ContainerStarted","Data":"4fa4a2e1be0c288ac624c8216d9c0bf3784613aec838a9658ab104707ff5d715"}
Jul 27 17:33:24.082139 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:24.082098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" event={"ID":"41af1395-97b8-4816-8f63-5e2e8e078611","Type":"ContainerStarted","Data":"262bd51b5820d67d40931d34291b60b1907555acedba6c6bea1a6666fb950262"}
Jul 27 17:33:24.082354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:24.082154    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:33:24.099771 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:24.099713    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" podStartSLOduration=1.099692154 podStartE2EDuration="1.099692154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:33:24.099339288 +0000 UTC m=+1912.684700844" watchObservedRunningTime="2026-07-27 17:33:24.099692154 +0000 UTC m=+1912.685053711"
Jul 27 17:33:55.089028 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:33:55.088997    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:34:07.694951 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.694891    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"]
Jul 27 17:34:07.698235 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.698217    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.701108 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.701086    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 27 17:34:07.707062 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.707038    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"]
Jul 27 17:34:07.871419 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.871384    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.871650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.871447    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6c1571-de34-4009-b5ab-08d3dacc4146-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.871650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.871530    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.871650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.871554    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.871650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.871576    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfh5v\" (UniqueName: \"kubernetes.io/projected/5c6c1571-de34-4009-b5ab-08d3dacc4146-kube-api-access-jfh5v\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.871650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.871615    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.972595 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.972564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6c1571-de34-4009-b5ab-08d3dacc4146-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.972788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.972615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.972788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.972654    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.972788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.972686    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfh5v\" (UniqueName: \"kubernetes.io/projected/5c6c1571-de34-4009-b5ab-08d3dacc4146-kube-api-access-jfh5v\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.972788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.972734    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.972788 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.972774    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.973147 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.973114    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.973269 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.973151    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.973269 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.973181    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.975192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.975171    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.975446 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.975427    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6c1571-de34-4009-b5ab-08d3dacc4146-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:07.983364 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:07.983340    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfh5v\" (UniqueName: \"kubernetes.io/projected/5c6c1571-de34-4009-b5ab-08d3dacc4146-kube-api-access-jfh5v\") pod \"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:08.009670 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:08.009643    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:08.146873 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:08.146838    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"]
Jul 27 17:34:08.150151 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:34:08.150121    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6c1571_de34_4009_b5ab_08d3dacc4146.slice/crio-0c40607f2ad34caafb3b4bc4913654b03209e3261507ddc652248c4db72faa30 WatchSource:0}: Error finding container 0c40607f2ad34caafb3b4bc4913654b03209e3261507ddc652248c4db72faa30: Status 404 returned error can't find the container with id 0c40607f2ad34caafb3b4bc4913654b03209e3261507ddc652248c4db72faa30
Jul 27 17:34:08.276434 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:08.276343    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" event={"ID":"5c6c1571-de34-4009-b5ab-08d3dacc4146","Type":"ContainerStarted","Data":"b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"}
Jul 27 17:34:08.276434 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:08.276400    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" event={"ID":"5c6c1571-de34-4009-b5ab-08d3dacc4146","Type":"ContainerStarted","Data":"0c40607f2ad34caafb3b4bc4913654b03209e3261507ddc652248c4db72faa30"}
Jul 27 17:34:08.294314 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:08.294261    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" podStartSLOduration=1.294244115 podStartE2EDuration="1.294244115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:34:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:34:08.292380261 +0000 UTC m=+1956.877741818" watchObservedRunningTime="2026-07-27 17:34:08.294244115 +0000 UTC m=+1956.879605670"
Jul 27 17:34:09.010801 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:09.010767    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:18.010223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:18.010173    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:18.022601 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:18.022577    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:18.332053 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:18.331963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:41.224201 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.224120    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"]
Jul 27 17:34:41.233430 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:34:41.233395    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:34:41.233430 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:34:41.233430 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:34:41.233430 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b" cmd=["/bin/sleep","15"]
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.233447    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" message="" err=<
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:34:41.233462    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerName="main" containerID="cri-o://b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"
Jul 27 17:34:41.233740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.233511    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerName="main" containerID="cri-o://b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b" gracePeriod=60
Jul 27 17:34:41.321597 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.321551    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": dial tcp 10.134.0.50:8000: connect: connection refused"
Jul 27 17:34:41.581290 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.581265    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:41.682433 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682395    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6c1571-de34-4009-b5ab-08d3dacc4146-tls-certs\") pod \"5c6c1571-de34-4009-b5ab-08d3dacc4146\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") "
Jul 27 17:34:41.682433 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682433    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-home\") pod \"5c6c1571-de34-4009-b5ab-08d3dacc4146\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") "
Jul 27 17:34:41.682713 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682461    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfh5v\" (UniqueName: \"kubernetes.io/projected/5c6c1571-de34-4009-b5ab-08d3dacc4146-kube-api-access-jfh5v\") pod \"5c6c1571-de34-4009-b5ab-08d3dacc4146\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") "
Jul 27 17:34:41.682713 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682505    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-tmp-dir\") pod \"5c6c1571-de34-4009-b5ab-08d3dacc4146\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") "
Jul 27 17:34:41.682713 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682520    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-dshm\") pod \"5c6c1571-de34-4009-b5ab-08d3dacc4146\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") "
Jul 27 17:34:41.682713 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682577    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-model-cache\") pod \"5c6c1571-de34-4009-b5ab-08d3dacc4146\" (UID: \"5c6c1571-de34-4009-b5ab-08d3dacc4146\") "
Jul 27 17:34:41.682908 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682779    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5c6c1571-de34-4009-b5ab-08d3dacc4146" (UID: "5c6c1571-de34-4009-b5ab-08d3dacc4146"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:34:41.682908 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682833    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-home" (OuterVolumeSpecName: "home") pod "5c6c1571-de34-4009-b5ab-08d3dacc4146" (UID: "5c6c1571-de34-4009-b5ab-08d3dacc4146"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:34:41.683024 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.682970    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-model-cache" (OuterVolumeSpecName: "model-cache") pod "5c6c1571-de34-4009-b5ab-08d3dacc4146" (UID: "5c6c1571-de34-4009-b5ab-08d3dacc4146"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:34:41.685080 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.685027    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6c1571-de34-4009-b5ab-08d3dacc4146-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5c6c1571-de34-4009-b5ab-08d3dacc4146" (UID: "5c6c1571-de34-4009-b5ab-08d3dacc4146"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:34:41.685228 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.685097    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-dshm" (OuterVolumeSpecName: "dshm") pod "5c6c1571-de34-4009-b5ab-08d3dacc4146" (UID: "5c6c1571-de34-4009-b5ab-08d3dacc4146"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:34:41.685668 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.685640    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6c1571-de34-4009-b5ab-08d3dacc4146-kube-api-access-jfh5v" (OuterVolumeSpecName: "kube-api-access-jfh5v") pod "5c6c1571-de34-4009-b5ab-08d3dacc4146" (UID: "5c6c1571-de34-4009-b5ab-08d3dacc4146"). InnerVolumeSpecName "kube-api-access-jfh5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:34:41.783765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.783685    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c6c1571-de34-4009-b5ab-08d3dacc4146-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:34:41.783765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.783711    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:34:41.783765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.783721    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfh5v\" (UniqueName: \"kubernetes.io/projected/5c6c1571-de34-4009-b5ab-08d3dacc4146-kube-api-access-jfh5v\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:34:41.783765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.783731    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:34:41.783765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.783742    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:34:41.783765 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:41.783749    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c6c1571-de34-4009-b5ab-08d3dacc4146-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:34:42.435144 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.435110    2579 generic.go:356] "Generic (PLEG): container finished" podID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerID="b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b" exitCode=0
Jul 27 17:34:42.435625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.435158    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"
Jul 27 17:34:42.435625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.435163    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" event={"ID":"5c6c1571-de34-4009-b5ab-08d3dacc4146","Type":"ContainerDied","Data":"b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"}
Jul 27 17:34:42.435625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.435203    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4" event={"ID":"5c6c1571-de34-4009-b5ab-08d3dacc4146","Type":"ContainerDied","Data":"0c40607f2ad34caafb3b4bc4913654b03209e3261507ddc652248c4db72faa30"}
Jul 27 17:34:42.435625 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.435225    2579 scope.go:117] "RemoveContainer" containerID="b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"
Jul 27 17:34:42.444146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.444127    2579 scope.go:117] "RemoveContainer" containerID="b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"
Jul 27 17:34:42.444429 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:34:42.444403    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b\": container with ID starting with b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b not found: ID does not exist" containerID="b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"
Jul 27 17:34:42.444549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.444439    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b"} err="failed to get container status \"b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b\": rpc error: code = NotFound desc = could not find container \"b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b\": container with ID starting with b24683846e54b3314b479bca075d287d8f2b1e7835ed9796180537369e294e4b not found: ID does not exist"
Jul 27 17:34:42.452349 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.452298    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"]
Jul 27 17:34:42.455246 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:42.455225    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-model-qwen2-5-0-5b-rout-a50492e9-kserve-5cf996c995xd6t4"]
Jul 27 17:34:43.892983 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:34:43.892946    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" path="/var/lib/kubelet/pods/5c6c1571-de34-4009-b5ab-08d3dacc4146/volumes"
Jul 27 17:35:01.360939 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:01.360903    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"]
Jul 27 17:35:05.089315 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:05.089276    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:35:15.088538 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:15.088498    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:35:16.380697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.380657    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" containerName="main" containerID="cri-o://4fa4a2e1be0c288ac624c8216d9c0bf3784613aec838a9658ab104707ff5d715" gracePeriod=45
Jul 27 17:35:16.593612 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.593576    2579 generic.go:356] "Generic (PLEG): container finished" podID="41af1395-97b8-4816-8f63-5e2e8e078611" containerID="4fa4a2e1be0c288ac624c8216d9c0bf3784613aec838a9658ab104707ff5d715" exitCode=0
Jul 27 17:35:16.593797 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.593631    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" event={"ID":"41af1395-97b8-4816-8f63-5e2e8e078611","Type":"ContainerDied","Data":"4fa4a2e1be0c288ac624c8216d9c0bf3784613aec838a9658ab104707ff5d715"}
Jul 27 17:35:16.640211 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.640150    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:35:16.666808 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.666777    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41af1395-97b8-4816-8f63-5e2e8e078611-tls-certs\") pod \"41af1395-97b8-4816-8f63-5e2e8e078611\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") "
Jul 27 17:35:16.666954 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.666868    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gk5rr\" (UniqueName: \"kubernetes.io/projected/41af1395-97b8-4816-8f63-5e2e8e078611-kube-api-access-gk5rr\") pod \"41af1395-97b8-4816-8f63-5e2e8e078611\" (UID: \"41af1395-97b8-4816-8f63-5e2e8e078611\") "
Jul 27 17:35:16.668998 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.668967    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41af1395-97b8-4816-8f63-5e2e8e078611-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41af1395-97b8-4816-8f63-5e2e8e078611" (UID: "41af1395-97b8-4816-8f63-5e2e8e078611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:35:16.669112 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.669049    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41af1395-97b8-4816-8f63-5e2e8e078611-kube-api-access-gk5rr" (OuterVolumeSpecName: "kube-api-access-gk5rr") pod "41af1395-97b8-4816-8f63-5e2e8e078611" (UID: "41af1395-97b8-4816-8f63-5e2e8e078611"). InnerVolumeSpecName "kube-api-access-gk5rr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:35:16.767674 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.767612    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gk5rr\" (UniqueName: \"kubernetes.io/projected/41af1395-97b8-4816-8f63-5e2e8e078611-kube-api-access-gk5rr\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:35:16.767674 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:16.767664    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41af1395-97b8-4816-8f63-5e2e8e078611-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:35:17.599347 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:17.599315    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"
Jul 27 17:35:17.599891 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:17.599350    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q" event={"ID":"41af1395-97b8-4816-8f63-5e2e8e078611","Type":"ContainerDied","Data":"262bd51b5820d67d40931d34291b60b1907555acedba6c6bea1a6666fb950262"}
Jul 27 17:35:17.599891 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:17.599409    2579 scope.go:117] "RemoveContainer" containerID="4fa4a2e1be0c288ac624c8216d9c0bf3784613aec838a9658ab104707ff5d715"
Jul 27 17:35:17.621304 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:17.621281    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"]
Jul 27 17:35:17.625819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:17.625799    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-schetld7q"]
Jul 27 17:35:17.892763 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:17.892684    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" path="/var/lib/kubelet/pods/41af1395-97b8-4816-8f63-5e2e8e078611/volumes"
Jul 27 17:35:41.032223 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032192    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"]
Jul 27 17:35:41.032701 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032652    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerName="main"
Jul 27 17:35:41.032701 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032671    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerName="main"
Jul 27 17:35:41.032819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032700    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" containerName="main"
Jul 27 17:35:41.032819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032710    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" containerName="main"
Jul 27 17:35:41.032819 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032809    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="41af1395-97b8-4816-8f63-5e2e8e078611" containerName="main"
Jul 27 17:35:41.032972 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.032821    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c6c1571-de34-4009-b5ab-08d3dacc4146" containerName="main"
Jul 27 17:35:41.035824 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.035802    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.040386 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.040165    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Jul 27 17:35:41.040386 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.040179    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-cstzn\""
Jul 27 17:35:41.040386 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.040165    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Jul 27 17:35:41.040386 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.040263    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-k9wdz\""
Jul 27 17:35:41.040386 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.040184    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Jul 27 17:35:41.046306 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.046283    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"]
Jul 27 17:35:41.163218 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.163182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbs5\" (UniqueName: \"kubernetes.io/projected/7de061de-40a9-4ea5-a7d2-c4940133f80d-kube-api-access-lkbs5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.163385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.163260    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de061de-40a9-4ea5-a7d2-c4940133f80d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.264654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.264613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbs5\" (UniqueName: \"kubernetes.io/projected/7de061de-40a9-4ea5-a7d2-c4940133f80d-kube-api-access-lkbs5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.264823 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.264681    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de061de-40a9-4ea5-a7d2-c4940133f80d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.267269 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.267234    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de061de-40a9-4ea5-a7d2-c4940133f80d-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.273135 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.273091    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbs5\" (UniqueName: \"kubernetes.io/projected/7de061de-40a9-4ea5-a7d2-c4940133f80d-kube-api-access-lkbs5\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.347554 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.347456    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.477448 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.477422    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"]
Jul 27 17:35:41.479668 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:35:41.479636    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de061de_40a9_4ea5_a7d2_c4940133f80d.slice/crio-4087d8141ed0dea06ea432b8160384674ca3eb4d822de144e3319d0bf2d0ca20 WatchSource:0}: Error finding container 4087d8141ed0dea06ea432b8160384674ca3eb4d822de144e3319d0bf2d0ca20: Status 404 returned error can't find the container with id 4087d8141ed0dea06ea432b8160384674ca3eb4d822de144e3319d0bf2d0ca20
Jul 27 17:35:41.701107 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.701010    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" event={"ID":"7de061de-40a9-4ea5-a7d2-c4940133f80d","Type":"ContainerStarted","Data":"f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052"}
Jul 27 17:35:41.701107 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.701072    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" event={"ID":"7de061de-40a9-4ea5-a7d2-c4940133f80d","Type":"ContainerStarted","Data":"4087d8141ed0dea06ea432b8160384674ca3eb4d822de144e3319d0bf2d0ca20"}
Jul 27 17:35:41.701303 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.701129    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:35:41.717869 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:35:41.717822    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" podStartSLOduration=0.717807001 podStartE2EDuration="717.807001ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:35:41.716870042 +0000 UTC m=+2050.302231599" watchObservedRunningTime="2026-07-27 17:35:41.717807001 +0000 UTC m=+2050.303168536"
Jul 27 17:36:12.707121 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:12.707087    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:36:24.593162 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.593127    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"]
Jul 27 17:36:24.596752 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.596729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.599310 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.599288    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Jul 27 17:36:24.605360 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.605334    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"]
Jul 27 17:36:24.628751 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.628720    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.628930 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.628755    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.628930 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.628778    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7sb\" (UniqueName: \"kubernetes.io/projected/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-kube-api-access-st7sb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.628930 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.628868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.629054 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.628931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.629054 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.628954    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730308 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730270    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730308 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730309    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730339    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730356    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7sb\" (UniqueName: \"kubernetes.io/projected/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-kube-api-access-st7sb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730602 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730405    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730762    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tmp-dir\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730825 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730760    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-home\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.730916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.730821    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-model-cache\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.732860 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.732834    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-dshm\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.733055 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.733040    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tls-certs\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.739032 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.739008    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7sb\" (UniqueName: \"kubernetes.io/projected/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-kube-api-access-st7sb\") pod \"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:24.908208 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:24.908117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:25.050920 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:25.050894    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"]
Jul 27 17:36:25.053291 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:36:25.053263    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edfc6d3_45f9_4b10_b2b9_1d9dd3a7a77e.slice/crio-46adb2386757d4bdb9f072997babc2d2d2d8f4cd6915eb074e98bf643a413a02 WatchSource:0}: Error finding container 46adb2386757d4bdb9f072997babc2d2d2d8f4cd6915eb074e98bf643a413a02: Status 404 returned error can't find the container with id 46adb2386757d4bdb9f072997babc2d2d2d8f4cd6915eb074e98bf643a413a02
Jul 27 17:36:25.900619 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:25.900584    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" event={"ID":"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e","Type":"ContainerStarted","Data":"adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"}
Jul 27 17:36:25.901018 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:25.900632    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" event={"ID":"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e","Type":"ContainerStarted","Data":"46adb2386757d4bdb9f072997babc2d2d2d8f4cd6915eb074e98bf643a413a02"}
Jul 27 17:36:25.908256 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:25.908234    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:25.918888 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:25.918833    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" podStartSLOduration=1.9188182980000001 podStartE2EDuration="1.918818298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:36:25.91689174 +0000 UTC m=+2094.502253308" watchObservedRunningTime="2026-07-27 17:36:25.918818298 +0000 UTC m=+2094.504179853"
Jul 27 17:36:31.887327 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:31.887298    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:36:31.900534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:31.900513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:36:34.909333 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:34.909270    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:34.921655 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:34.921630    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:34.949977 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:34.949945    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:57.999947 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:57.999909    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"]
Jul 27 17:36:58.010862 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:36:58.010821    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:36:58.010862 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:36:58.010862 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:36:58.010862 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318" cmd=["/bin/sleep","15"]
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.010874    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" message="" err=<
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:36:58.010886    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" podUID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" containerName="main" containerID="cri-o://adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"
Jul 27 17:36:58.011194 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.010942    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" podUID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" containerName="main" containerID="cri-o://adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318" gracePeriod=60
Jul 27 17:36:58.263250 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.263181    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:58.425642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.425613    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-dshm\") pod \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") "
Jul 27 17:36:58.425844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.425658    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7sb\" (UniqueName: \"kubernetes.io/projected/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-kube-api-access-st7sb\") pod \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") "
Jul 27 17:36:58.425844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.425682    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-model-cache\") pod \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") "
Jul 27 17:36:58.425844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.425709    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tmp-dir\") pod \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") "
Jul 27 17:36:58.425844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.425743    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-home\") pod \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") "
Jul 27 17:36:58.425844 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.425783    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tls-certs\") pod \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\" (UID: \"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e\") "
Jul 27 17:36:58.426179 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.426000    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" (UID: "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:58.426179 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.426043    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" (UID: "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:58.426179 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.426073    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:36:58.426179 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.426049    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-home" (OuterVolumeSpecName: "home") pod "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" (UID: "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:58.428221 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.428192    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" (UID: "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:36:58.428459 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.428441    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-dshm" (OuterVolumeSpecName: "dshm") pod "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" (UID: "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:36:58.428537 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.428467    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-kube-api-access-st7sb" (OuterVolumeSpecName: "kube-api-access-st7sb") pod "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" (UID: "2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e"). InnerVolumeSpecName "kube-api-access-st7sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:36:58.526707 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.526615    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:36:58.526707 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.526647    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7sb\" (UniqueName: \"kubernetes.io/projected/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-kube-api-access-st7sb\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:36:58.526707 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.526685    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:36:58.526707 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.526698    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:36:58.526707 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:58.526710    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:36:59.054324 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.054288    2579 generic.go:356] "Generic (PLEG): container finished" podID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" containerID="adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318" exitCode=0
Jul 27 17:36:59.054837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.054345    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"
Jul 27 17:36:59.054837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.054376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" event={"ID":"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e","Type":"ContainerDied","Data":"adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"}
Jul 27 17:36:59.054837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.054431    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq" event={"ID":"2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e","Type":"ContainerDied","Data":"46adb2386757d4bdb9f072997babc2d2d2d8f4cd6915eb074e98bf643a413a02"}
Jul 27 17:36:59.054837 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.054457    2579 scope.go:117] "RemoveContainer" containerID="adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"
Jul 27 17:36:59.064801 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.064786    2579 scope.go:117] "RemoveContainer" containerID="adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"
Jul 27 17:36:59.065079 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:36:59.065063    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318\": container with ID starting with adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318 not found: ID does not exist" containerID="adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"
Jul 27 17:36:59.065146 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.065086    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318"} err="failed to get container status \"adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318\": rpc error: code = NotFound desc = could not find container \"adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318\": container with ID starting with adf7b2a81450ecd31fdf787461ebf07c1c12c5abbb7a9a4dde73cbe934653318 not found: ID does not exist"
Jul 27 17:36:59.077181 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.077157    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"]
Jul 27 17:36:59.079576 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.079556    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-kserve-bb749b474-zbnkq"]
Jul 27 17:36:59.894307 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:36:59.894272    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" path="/var/lib/kubelet/pods/2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e/volumes"
Jul 27 17:37:18.133047 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:18.132994    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"]
Jul 27 17:37:22.706567 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:22.706464    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:37:32.707125 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:32.707028    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:37:33.154192 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.154104    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerName="main" containerID="cri-o://f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052" gracePeriod=45
Jul 27 17:37:33.403354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.403329    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:37:33.503279 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.503249    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lkbs5\" (UniqueName: \"kubernetes.io/projected/7de061de-40a9-4ea5-a7d2-c4940133f80d-kube-api-access-lkbs5\") pod \"7de061de-40a9-4ea5-a7d2-c4940133f80d\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") "
Jul 27 17:37:33.503438 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.503305    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de061de-40a9-4ea5-a7d2-c4940133f80d-tls-certs\") pod \"7de061de-40a9-4ea5-a7d2-c4940133f80d\" (UID: \"7de061de-40a9-4ea5-a7d2-c4940133f80d\") "
Jul 27 17:37:33.505644 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.505606    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de061de-40a9-4ea5-a7d2-c4940133f80d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7de061de-40a9-4ea5-a7d2-c4940133f80d" (UID: "7de061de-40a9-4ea5-a7d2-c4940133f80d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:37:33.505745 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.505636    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de061de-40a9-4ea5-a7d2-c4940133f80d-kube-api-access-lkbs5" (OuterVolumeSpecName: "kube-api-access-lkbs5") pod "7de061de-40a9-4ea5-a7d2-c4940133f80d" (UID: "7de061de-40a9-4ea5-a7d2-c4940133f80d"). InnerVolumeSpecName "kube-api-access-lkbs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:37:33.604879 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.604846    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkbs5\" (UniqueName: \"kubernetes.io/projected/7de061de-40a9-4ea5-a7d2-c4940133f80d-kube-api-access-lkbs5\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:37:33.604879 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:33.604873    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7de061de-40a9-4ea5-a7d2-c4940133f80d-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:37:34.216486 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.216441    2579 generic.go:356] "Generic (PLEG): container finished" podID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerID="f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052" exitCode=0
Jul 27 17:37:34.216922 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.216497    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"
Jul 27 17:37:34.216922 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.216504    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" event={"ID":"7de061de-40a9-4ea5-a7d2-c4940133f80d","Type":"ContainerDied","Data":"f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052"}
Jul 27 17:37:34.216922 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.216546    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt" event={"ID":"7de061de-40a9-4ea5-a7d2-c4940133f80d","Type":"ContainerDied","Data":"4087d8141ed0dea06ea432b8160384674ca3eb4d822de144e3319d0bf2d0ca20"}
Jul 27 17:37:34.216922 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.216568    2579 scope.go:117] "RemoveContainer" containerID="f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052"
Jul 27 17:37:34.224559 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.224542    2579 scope.go:117] "RemoveContainer" containerID="f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052"
Jul 27 17:37:34.224838 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:37:34.224820    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052\": container with ID starting with f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052 not found: ID does not exist" containerID="f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052"
Jul 27 17:37:34.224898 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.224847    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052"} err="failed to get container status \"f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052\": rpc error: code = NotFound desc = could not find container \"f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052\": container with ID starting with f5f64b3fabdfcda6f7a3f5c0b632d0f4c52a65c25ea4faee135270d603285052 not found: ID does not exist"
Jul 27 17:37:34.235808 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.235779    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"]
Jul 27 17:37:34.239083 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:34.239060    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-schegk5kt"]
Jul 27 17:37:35.893032 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:35.892993    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" path="/var/lib/kubelet/pods/7de061de-40a9-4ea5-a7d2-c4940133f80d/volumes"
Jul 27 17:37:41.617070 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617033    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"]
Jul 27 17:37:41.617596 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617576    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" containerName="main"
Jul 27 17:37:41.617663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617601    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" containerName="main"
Jul 27 17:37:41.617663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617616    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerName="main"
Jul 27 17:37:41.617663 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617624    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerName="main"
Jul 27 17:37:41.617770 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617732    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edfc6d3-45f9-4b10-b2b9-1d9dd3a7a77e" containerName="main"
Jul 27 17:37:41.617770 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.617747    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7de061de-40a9-4ea5-a7d2-c4940133f80d" containerName="main"
Jul 27 17:37:41.620975 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.620957    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.624585 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.624559    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-n2xlj\""
Jul 27 17:37:41.624723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.624659    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 27 17:37:41.624723 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.624680    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Jul 27 17:37:41.624836 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.624821    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Jul 27 17:37:41.624920 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.624904    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-stljr\""
Jul 27 17:37:41.628611 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.628590    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"]
Jul 27 17:37:41.674135 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.674106    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm5mz\" (UniqueName: \"kubernetes.io/projected/f3b54f75-fa93-4ee6-9af5-d927920148fd-kube-api-access-lm5mz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.674287 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.674175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b54f75-fa93-4ee6-9af5-d927920148fd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.775350 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.775312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b54f75-fa93-4ee6-9af5-d927920148fd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.775541 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.775375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm5mz\" (UniqueName: \"kubernetes.io/projected/f3b54f75-fa93-4ee6-9af5-d927920148fd-kube-api-access-lm5mz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.777978 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.777958    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b54f75-fa93-4ee6-9af5-d927920148fd-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.783495 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.783452    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm5mz\" (UniqueName: \"kubernetes.io/projected/f3b54f75-fa93-4ee6-9af5-d927920148fd-kube-api-access-lm5mz\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:41.932275 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:41.932201    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:42.067413 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:42.067362    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"]
Jul 27 17:37:42.069888 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:37:42.069859    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b54f75_fa93_4ee6_9af5_d927920148fd.slice/crio-d9181a56c8744546558e06d94db42b5385c9de9a757a815a38046c2400d8bf77 WatchSource:0}: Error finding container d9181a56c8744546558e06d94db42b5385c9de9a757a815a38046c2400d8bf77: Status 404 returned error can't find the container with id d9181a56c8744546558e06d94db42b5385c9de9a757a815a38046c2400d8bf77
Jul 27 17:37:42.260735 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:42.260696    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" event={"ID":"f3b54f75-fa93-4ee6-9af5-d927920148fd","Type":"ContainerStarted","Data":"c3c7defc6ba153d7e0b6458ade154c5a873f9ca506e65ff5443bc51532ee79a6"}
Jul 27 17:37:42.260928 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:42.260761    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" event={"ID":"f3b54f75-fa93-4ee6-9af5-d927920148fd","Type":"ContainerStarted","Data":"d9181a56c8744546558e06d94db42b5385c9de9a757a815a38046c2400d8bf77"}
Jul 27 17:37:42.260928 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:42.260789    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:37:42.277791 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:37:42.277744    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" podStartSLOduration=1.277727257 podStartE2EDuration="1.277727257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:37:42.277106744 +0000 UTC m=+2170.862468301" watchObservedRunningTime="2026-07-27 17:37:42.277727257 +0000 UTC m=+2170.863088812"
Jul 27 17:38:13.267746 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:38:13.267715    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:39:37.618992 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:37.618960    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"]
Jul 27 17:39:43.267893 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:43.267846    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" podUID="f3b54f75-fa93-4ee6-9af5-d927920148fd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:39:52.638679 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.638622    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" podUID="f3b54f75-fa93-4ee6-9af5-d927920148fd" containerName="main" containerID="cri-o://c3c7defc6ba153d7e0b6458ade154c5a873f9ca506e65ff5443bc51532ee79a6" gracePeriod=45
Jul 27 17:39:52.838179 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.838142    2579 generic.go:356] "Generic (PLEG): container finished" podID="f3b54f75-fa93-4ee6-9af5-d927920148fd" containerID="c3c7defc6ba153d7e0b6458ade154c5a873f9ca506e65ff5443bc51532ee79a6" exitCode=0
Jul 27 17:39:52.838332 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.838181    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" event={"ID":"f3b54f75-fa93-4ee6-9af5-d927920148fd","Type":"ContainerDied","Data":"c3c7defc6ba153d7e0b6458ade154c5a873f9ca506e65ff5443bc51532ee79a6"}
Jul 27 17:39:52.891029 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.890960    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:39:52.979743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.979718    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm5mz\" (UniqueName: \"kubernetes.io/projected/f3b54f75-fa93-4ee6-9af5-d927920148fd-kube-api-access-lm5mz\") pod \"f3b54f75-fa93-4ee6-9af5-d927920148fd\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") "
Jul 27 17:39:52.979895 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.979788    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b54f75-fa93-4ee6-9af5-d927920148fd-tls-certs\") pod \"f3b54f75-fa93-4ee6-9af5-d927920148fd\" (UID: \"f3b54f75-fa93-4ee6-9af5-d927920148fd\") "
Jul 27 17:39:52.982029 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.981992    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3b54f75-fa93-4ee6-9af5-d927920148fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f3b54f75-fa93-4ee6-9af5-d927920148fd" (UID: "f3b54f75-fa93-4ee6-9af5-d927920148fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:39:52.982152 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:52.982037    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b54f75-fa93-4ee6-9af5-d927920148fd-kube-api-access-lm5mz" (OuterVolumeSpecName: "kube-api-access-lm5mz") pod "f3b54f75-fa93-4ee6-9af5-d927920148fd" (UID: "f3b54f75-fa93-4ee6-9af5-d927920148fd"). InnerVolumeSpecName "kube-api-access-lm5mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:39:53.080916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.080879    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm5mz\" (UniqueName: \"kubernetes.io/projected/f3b54f75-fa93-4ee6-9af5-d927920148fd-kube-api-access-lm5mz\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.080916 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.080909    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3b54f75-fa93-4ee6-9af5-d927920148fd-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:39:53.844137 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.844102    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2" event={"ID":"f3b54f75-fa93-4ee6-9af5-d927920148fd","Type":"ContainerDied","Data":"d9181a56c8744546558e06d94db42b5385c9de9a757a815a38046c2400d8bf77"}
Jul 27 17:39:53.844137 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.844131    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"
Jul 27 17:39:53.844701 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.844161    2579 scope.go:117] "RemoveContainer" containerID="c3c7defc6ba153d7e0b6458ade154c5a873f9ca506e65ff5443bc51532ee79a6"
Jul 27 17:39:53.865489 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.865455    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"]
Jul 27 17:39:53.869496 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.869461    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-scheh9tw2"]
Jul 27 17:39:53.892631 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:39:53.892605    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3b54f75-fa93-4ee6-9af5-d927920148fd" path="/var/lib/kubelet/pods/f3b54f75-fa93-4ee6-9af5-d927920148fd/volumes"
Jul 27 17:40:05.288608 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.288573    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"]
Jul 27 17:40:05.289087 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.288888    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3b54f75-fa93-4ee6-9af5-d927920148fd" containerName="main"
Jul 27 17:40:05.289087 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.288900    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3b54f75-fa93-4ee6-9af5-d927920148fd" containerName="main"
Jul 27 17:40:05.289087 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.288961    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3b54f75-fa93-4ee6-9af5-d927920148fd" containerName="main"
Jul 27 17:40:05.293300 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.293265    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.298113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.297925    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Jul 27 17:40:05.298113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.298002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-pbvkd\""
Jul 27 17:40:05.298113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.298009    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 27 17:40:05.298113 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.298027    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Jul 27 17:40:05.298403 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.298190    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-ntxbb\""
Jul 27 17:40:05.306466 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.306444    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"]
Jul 27 17:40:05.481555 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.481521    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92ntz\" (UniqueName: \"kubernetes.io/projected/b1364b86-cbd2-47b9-bb11-c9e3429750c2-kube-api-access-92ntz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.481749 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.481567    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1364b86-cbd2-47b9-bb11-c9e3429750c2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.582364 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.582273    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92ntz\" (UniqueName: \"kubernetes.io/projected/b1364b86-cbd2-47b9-bb11-c9e3429750c2-kube-api-access-92ntz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.582364 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.582317    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1364b86-cbd2-47b9-bb11-c9e3429750c2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.585031 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.585003    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1364b86-cbd2-47b9-bb11-c9e3429750c2-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.592198 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.592173    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92ntz\" (UniqueName: \"kubernetes.io/projected/b1364b86-cbd2-47b9-bb11-c9e3429750c2-kube-api-access-92ntz\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.604946 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.604921    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.738703 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.738678    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"]
Jul 27 17:40:05.740940 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:40:05.740910    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1364b86_cbd2_47b9_bb11_c9e3429750c2.slice/crio-22ef46c1bef3683d575074ddc560bbd4bce8ab437c3e2d46f79b42652dc2f23c WatchSource:0}: Error finding container 22ef46c1bef3683d575074ddc560bbd4bce8ab437c3e2d46f79b42652dc2f23c: Status 404 returned error can't find the container with id 22ef46c1bef3683d575074ddc560bbd4bce8ab437c3e2d46f79b42652dc2f23c
Jul 27 17:40:05.896460 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.896376    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" event={"ID":"b1364b86-cbd2-47b9-bb11-c9e3429750c2","Type":"ContainerStarted","Data":"a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc"}
Jul 27 17:40:05.896460 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.896423    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" event={"ID":"b1364b86-cbd2-47b9-bb11-c9e3429750c2","Type":"ContainerStarted","Data":"22ef46c1bef3683d575074ddc560bbd4bce8ab437c3e2d46f79b42652dc2f23c"}
Jul 27 17:40:05.896677 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.896552    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:40:05.914356 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:05.914297    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" podStartSLOduration=0.914277753 podStartE2EDuration="914.277753ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:40:05.914059433 +0000 UTC m=+2314.499420990" watchObservedRunningTime="2026-07-27 17:40:05.914277753 +0000 UTC m=+2314.499639310"
Jul 27 17:40:36.902703 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:40:36.902674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:41:00.024910 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.024876    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"]
Jul 27 17:41:00.035745 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:41:00.035703    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:41:00.035745 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:41:00.035745 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:41:00.035745 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11" cmd=["/bin/sleep","15"]
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.035767    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" message="" err=<
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:41:00.035784    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerName="main" containerID="cri-o://1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"
Jul 27 17:41:00.036061 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.035835    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerName="main" containerID="cri-o://1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11" gracePeriod=60
Jul 27 17:41:00.262394 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.262351    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": dial tcp 10.134.0.42:8000: connect: connection refused"
Jul 27 17:41:00.786092 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.786064    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:41:00.933312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933199    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-dshm\") pod \"50914253-8935-4351-a8f8-eb12fe2cc6e4\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") "
Jul 27 17:41:00.933312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933276    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-model-cache\") pod \"50914253-8935-4351-a8f8-eb12fe2cc6e4\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") "
Jul 27 17:41:00.933312 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933303    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-home\") pod \"50914253-8935-4351-a8f8-eb12fe2cc6e4\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") "
Jul 27 17:41:00.933584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933330    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50914253-8935-4351-a8f8-eb12fe2cc6e4-tls-certs\") pod \"50914253-8935-4351-a8f8-eb12fe2cc6e4\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") "
Jul 27 17:41:00.933584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933379    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-tmp-dir\") pod \"50914253-8935-4351-a8f8-eb12fe2cc6e4\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") "
Jul 27 17:41:00.933584 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933535    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlf2v\" (UniqueName: \"kubernetes.io/projected/50914253-8935-4351-a8f8-eb12fe2cc6e4-kube-api-access-vlf2v\") pod \"50914253-8935-4351-a8f8-eb12fe2cc6e4\" (UID: \"50914253-8935-4351-a8f8-eb12fe2cc6e4\") "
Jul 27 17:41:00.933743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933589    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "50914253-8935-4351-a8f8-eb12fe2cc6e4" (UID: "50914253-8935-4351-a8f8-eb12fe2cc6e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:41:00.933743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933645    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50914253-8935-4351-a8f8-eb12fe2cc6e4" (UID: "50914253-8935-4351-a8f8-eb12fe2cc6e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:41:00.933743 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933656    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-home" (OuterVolumeSpecName: "home") pod "50914253-8935-4351-a8f8-eb12fe2cc6e4" (UID: "50914253-8935-4351-a8f8-eb12fe2cc6e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:41:00.933931 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933765    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:41:00.933931 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933777    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:41:00.933931 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.933786    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:41:00.935710 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.935681    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-dshm" (OuterVolumeSpecName: "dshm") pod "50914253-8935-4351-a8f8-eb12fe2cc6e4" (UID: "50914253-8935-4351-a8f8-eb12fe2cc6e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:41:00.935811 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.935766    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50914253-8935-4351-a8f8-eb12fe2cc6e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50914253-8935-4351-a8f8-eb12fe2cc6e4" (UID: "50914253-8935-4351-a8f8-eb12fe2cc6e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:41:00.935865 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:00.935845    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50914253-8935-4351-a8f8-eb12fe2cc6e4-kube-api-access-vlf2v" (OuterVolumeSpecName: "kube-api-access-vlf2v") pod "50914253-8935-4351-a8f8-eb12fe2cc6e4" (UID: "50914253-8935-4351-a8f8-eb12fe2cc6e4"). InnerVolumeSpecName "kube-api-access-vlf2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:41:01.035199 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.035158    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50914253-8935-4351-a8f8-eb12fe2cc6e4-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:41:01.035199 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.035190    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlf2v\" (UniqueName: \"kubernetes.io/projected/50914253-8935-4351-a8f8-eb12fe2cc6e4-kube-api-access-vlf2v\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:41:01.035199 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.035205    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50914253-8935-4351-a8f8-eb12fe2cc6e4-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:41:01.125796 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.125760    2579 generic.go:356] "Generic (PLEG): container finished" podID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerID="1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11" exitCode=0
Jul 27 17:41:01.125984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.125806    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"
Jul 27 17:41:01.125984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.125845    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" event={"ID":"50914253-8935-4351-a8f8-eb12fe2cc6e4","Type":"ContainerDied","Data":"1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"}
Jul 27 17:41:01.125984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.125920    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs" event={"ID":"50914253-8935-4351-a8f8-eb12fe2cc6e4","Type":"ContainerDied","Data":"a84fdf4a6b8a8bc049a17835b614506fd175a0b1fa22387557f4b9f3c5abbc55"}
Jul 27 17:41:01.125984 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.125944    2579 scope.go:117] "RemoveContainer" containerID="1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"
Jul 27 17:41:01.134851 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.134834    2579 scope.go:117] "RemoveContainer" containerID="1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"
Jul 27 17:41:01.135103 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:41:01.135086    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11\": container with ID starting with 1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11 not found: ID does not exist" containerID="1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"
Jul 27 17:41:01.135148 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.135111    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11"} err="failed to get container status \"1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11\": rpc error: code = NotFound desc = could not find container \"1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11\": container with ID starting with 1db517db7fa49c292a1ff57ad8228526f4a4d216340776927d31f1a99ce9cc11 not found: ID does not exist"
Jul 27 17:41:01.145718 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.145696    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"]
Jul 27 17:41:01.149697 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.149677    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-deployment-9ba6f3f4/autoscale-hpa-deploy-kserve-66746bd8b9-crcfs"]
Jul 27 17:41:01.893326 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:01.893282    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" path="/var/lib/kubelet/pods/50914253-8935-4351-a8f8-eb12fe2cc6e4/volumes"
Jul 27 17:41:31.912359 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:31.912330    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:41:31.928744 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:31.928719    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:41:39.792290 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.792247    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"]
Jul 27 17:41:39.792740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.792643    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerName="main"
Jul 27 17:41:39.792740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.792661    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerName="main"
Jul 27 17:41:39.792740 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.792714    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50914253-8935-4351-a8f8-eb12fe2cc6e4" containerName="main"
Jul 27 17:41:39.795730 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.795711    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.799522 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.799484    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"autoscale-keda-deploy-kserve-self-signed-certs\""
Jul 27 17:41:39.799522 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.799503    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"default-dockercfg-bv4z2\""
Jul 27 17:41:39.799698 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.799506    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"openshift-service-ca.crt\""
Jul 27 17:41:39.799698 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.799503    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-deployment-b2150d0b\"/\"kube-root-ca.crt\""
Jul 27 17:41:39.804183 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.804162    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"]
Jul 27 17:41:39.846900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.846861    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr77\" (UniqueName: \"kubernetes.io/projected/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-kube-api-access-vsr77\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.846900 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.846905    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.847101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.846928    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.847101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.846951    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.847101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.846994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.847101 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.847013    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948318    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr77\" (UniqueName: \"kubernetes.io/projected/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-kube-api-access-vsr77\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948425    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948607 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948450    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-home\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948885    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tmp-dir\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.948949 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.948927    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-model-cache\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.950864 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.950841    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-dshm\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.951173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.951147    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tls-certs\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:39.955845 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:39.955826    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr77\" (UniqueName: \"kubernetes.io/projected/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-kube-api-access-vsr77\") pod \"autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") " pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:40.107359 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:40.107264    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:40.245501 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:40.245439    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"]
Jul 27 17:41:40.248613 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:41:40.248579    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fb8aee_0cd9_48b3_bd26_b59358ff12f3.slice/crio-03551959aa0b43b29d1c7390eb5b20a2bccdfdf1f83c1528c2ef24190f858945 WatchSource:0}: Error finding container 03551959aa0b43b29d1c7390eb5b20a2bccdfdf1f83c1528c2ef24190f858945: Status 404 returned error can't find the container with id 03551959aa0b43b29d1c7390eb5b20a2bccdfdf1f83c1528c2ef24190f858945
Jul 27 17:41:40.302546 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:40.302513    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" event={"ID":"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3","Type":"ContainerStarted","Data":"03551959aa0b43b29d1c7390eb5b20a2bccdfdf1f83c1528c2ef24190f858945"}
Jul 27 17:41:41.308143 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:41.308109    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" event={"ID":"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3","Type":"ContainerStarted","Data":"791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"}
Jul 27 17:41:41.329557 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:41.329506    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" podStartSLOduration=2.329489608 podStartE2EDuration="2.329489608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:41:41.327861978 +0000 UTC m=+2409.913223549" watchObservedRunningTime="2026-07-27 17:41:41.329489608 +0000 UTC m=+2409.914851158"
Jul 27 17:41:42.108455 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:42.108410    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:50.108084 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:50.108042    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:50.120767 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:50.120741    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:50.359385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:50.359297    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:41:55.392639 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:55.392551    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"]
Jul 27 17:41:56.901974 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:41:56.901927    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:42:06.901771 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:06.901728    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:42:10.412380 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.412337    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerName="main" containerID="cri-o://a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc" gracePeriod=45
Jul 27 17:42:10.668392 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.668331    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:42:10.821037 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.820996    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-92ntz\" (UniqueName: \"kubernetes.io/projected/b1364b86-cbd2-47b9-bb11-c9e3429750c2-kube-api-access-92ntz\") pod \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") "
Jul 27 17:42:10.821213 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.821078    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1364b86-cbd2-47b9-bb11-c9e3429750c2-tls-certs\") pod \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\" (UID: \"b1364b86-cbd2-47b9-bb11-c9e3429750c2\") "
Jul 27 17:42:10.823457 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.823418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1364b86-cbd2-47b9-bb11-c9e3429750c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1364b86-cbd2-47b9-bb11-c9e3429750c2" (UID: "b1364b86-cbd2-47b9-bb11-c9e3429750c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:42:10.823457 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.823429    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1364b86-cbd2-47b9-bb11-c9e3429750c2-kube-api-access-92ntz" (OuterVolumeSpecName: "kube-api-access-92ntz") pod "b1364b86-cbd2-47b9-bb11-c9e3429750c2" (UID: "b1364b86-cbd2-47b9-bb11-c9e3429750c2"). InnerVolumeSpecName "kube-api-access-92ntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:42:10.921897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.921803    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92ntz\" (UniqueName: \"kubernetes.io/projected/b1364b86-cbd2-47b9-bb11-c9e3429750c2-kube-api-access-92ntz\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:42:10.921897 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:10.921842    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1364b86-cbd2-47b9-bb11-c9e3429750c2-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:42:11.449904 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.449866    2579 generic.go:356] "Generic (PLEG): container finished" podID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerID="a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc" exitCode=0
Jul 27 17:42:11.450549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.449921    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"
Jul 27 17:42:11.450549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.449957    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" event={"ID":"b1364b86-cbd2-47b9-bb11-c9e3429750c2","Type":"ContainerDied","Data":"a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc"}
Jul 27 17:42:11.450549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.450007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579" event={"ID":"b1364b86-cbd2-47b9-bb11-c9e3429750c2","Type":"ContainerDied","Data":"22ef46c1bef3683d575074ddc560bbd4bce8ab437c3e2d46f79b42652dc2f23c"}
Jul 27 17:42:11.450549 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.450028    2579 scope.go:117] "RemoveContainer" containerID="a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc"
Jul 27 17:42:11.459755 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.459734    2579 scope.go:117] "RemoveContainer" containerID="a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc"
Jul 27 17:42:11.460043 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:42:11.460022    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc\": container with ID starting with a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc not found: ID does not exist" containerID="a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc"
Jul 27 17:42:11.460108 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.460053    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc"} err="failed to get container status \"a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc\": rpc error: code = NotFound desc = could not find container \"a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc\": container with ID starting with a87ad6cb595a947f506c62acd36379a35c90cf5c6ab8cd31f488bfb1810dfadc not found: ID does not exist"
Jul 27 17:42:11.471881 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.471849    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"]
Jul 27 17:42:11.475498 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.475457    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-scheh9579"]
Jul 27 17:42:11.893020 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:11.892975    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" path="/var/lib/kubelet/pods/b1364b86-cbd2-47b9-bb11-c9e3429750c2/volumes"
Jul 27 17:42:57.434982 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.434942    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"]
Jul 27 17:42:57.435616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.435314    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerName="main"
Jul 27 17:42:57.435616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.435326    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerName="main"
Jul 27 17:42:57.435616 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.435403    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1364b86-cbd2-47b9-bb11-c9e3429750c2" containerName="main"
Jul 27 17:42:57.438410 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.438389    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.442025 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.442001    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 27 17:42:57.442170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.442058    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"kube-root-ca.crt\""
Jul 27 17:42:57.442170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.442002    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"llmisvc-model-pvc-router-manage-59b9d263-epp-sa-dockercfg-p5s4c\""
Jul 27 17:42:57.442170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.442077    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"openshift-service-ca.crt\""
Jul 27 17:42:57.442170 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.442094    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cc472ec3\"/\"default-dockercfg-v5mkm\""
Jul 27 17:42:57.447906 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.447879    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"]
Jul 27 17:42:57.506804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.506767    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66sjc\" (UniqueName: \"kubernetes.io/projected/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-kube-api-access-66sjc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.506973 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.506819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.608066 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.608032    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66sjc\" (UniqueName: \"kubernetes.io/projected/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-kube-api-access-66sjc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.608222 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.608079    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.610644 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.610620    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-tls-certs\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.616931 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.616905    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66sjc\" (UniqueName: \"kubernetes.io/projected/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-kube-api-access-66sjc\") pod \"llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") " pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.749735 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.749706    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:57.883334 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:57.883307    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"]
Jul 27 17:42:57.885937 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:42:57.885905    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baf6486_41d8_4f73_8ce3_fde381e3d2ff.slice/crio-21499cf70063af8e77c80dc50dc2657bf0684f94e77ae32bc0d4dc891b9b27e0 WatchSource:0}: Error finding container 21499cf70063af8e77c80dc50dc2657bf0684f94e77ae32bc0d4dc891b9b27e0: Status 404 returned error can't find the container with id 21499cf70063af8e77c80dc50dc2657bf0684f94e77ae32bc0d4dc891b9b27e0
Jul 27 17:42:58.665947 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:58.665909    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" event={"ID":"0baf6486-41d8-4f73-8ce3-fde381e3d2ff","Type":"ContainerStarted","Data":"2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438"}
Jul 27 17:42:58.666354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:58.665973    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" event={"ID":"0baf6486-41d8-4f73-8ce3-fde381e3d2ff","Type":"ContainerStarted","Data":"21499cf70063af8e77c80dc50dc2657bf0684f94e77ae32bc0d4dc891b9b27e0"}
Jul 27 17:42:58.666354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:58.666002    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:42:58.683115 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:42:58.683064    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" podStartSLOduration=1.683049528 podStartE2EDuration="1.683049528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:42:58.68164355 +0000 UTC m=+2487.267005104" watchObservedRunningTime="2026-07-27 17:42:58.683049528 +0000 UTC m=+2487.268411083"
Jul 27 17:43:29.672444 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:43:29.672370    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:45:11.897776 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:11.897742    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"]
Jul 27 17:45:19.673624 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:19.673575    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" podUID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 17:45:26.917346 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:26.917305    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" podUID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" containerName="main" containerID="cri-o://2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438" gracePeriod=45
Jul 27 17:45:27.168220 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.168146    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:45:27.303903 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.303866    2579 generic.go:356] "Generic (PLEG): container finished" podID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" containerID="2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438" exitCode=0
Jul 27 17:45:27.303903 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.303907    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"
Jul 27 17:45:27.304130 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.303931    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" event={"ID":"0baf6486-41d8-4f73-8ce3-fde381e3d2ff","Type":"ContainerDied","Data":"2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438"}
Jul 27 17:45:27.304130 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.304007    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct" event={"ID":"0baf6486-41d8-4f73-8ce3-fde381e3d2ff","Type":"ContainerDied","Data":"21499cf70063af8e77c80dc50dc2657bf0684f94e77ae32bc0d4dc891b9b27e0"}
Jul 27 17:45:27.304130 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.304032    2579 scope.go:117] "RemoveContainer" containerID="2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438"
Jul 27 17:45:27.312246 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.312227    2579 scope.go:117] "RemoveContainer" containerID="2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438"
Jul 27 17:45:27.312563 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:45:27.312543    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438\": container with ID starting with 2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438 not found: ID does not exist" containerID="2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438"
Jul 27 17:45:27.312614 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.312573    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438"} err="failed to get container status \"2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438\": rpc error: code = NotFound desc = could not find container \"2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438\": container with ID starting with 2bac90ce89c9a9f36faf00f91bcf06cd70ac2bbec13fb9977bb247ea1afbd438 not found: ID does not exist"
Jul 27 17:45:27.347032 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.346998    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-tls-certs\") pod \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") "
Jul 27 17:45:27.347186 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.347043    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66sjc\" (UniqueName: \"kubernetes.io/projected/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-kube-api-access-66sjc\") pod \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\" (UID: \"0baf6486-41d8-4f73-8ce3-fde381e3d2ff\") "
Jul 27 17:45:27.349449 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.349416    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0baf6486-41d8-4f73-8ce3-fde381e3d2ff" (UID: "0baf6486-41d8-4f73-8ce3-fde381e3d2ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:45:27.349449 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.349418    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-kube-api-access-66sjc" (OuterVolumeSpecName: "kube-api-access-66sjc") pod "0baf6486-41d8-4f73-8ce3-fde381e3d2ff" (UID: "0baf6486-41d8-4f73-8ce3-fde381e3d2ff"). InnerVolumeSpecName "kube-api-access-66sjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:45:27.447991 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.447892    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.447991 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.447929    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66sjc\" (UniqueName: \"kubernetes.io/projected/0baf6486-41d8-4f73-8ce3-fde381e3d2ff-kube-api-access-66sjc\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:45:27.626768 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.626724    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"]
Jul 27 17:45:27.632247 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.632205    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cc472ec3/llmisvc8bf079eb6eda4debfb4ef5bb7817824c-kserve-router-sche5wdct"]
Jul 27 17:45:27.893317 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:45:27.893278    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" path="/var/lib/kubelet/pods/0baf6486-41d8-4f73-8ce3-fde381e3d2ff/volumes"
Jul 27 17:46:16.037846 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.037810    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"]
Jul 27 17:46:16.038294 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.038267    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" containerName="main"
Jul 27 17:46:16.038294 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.038282    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" containerName="main"
Jul 27 17:46:16.038385 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.038374    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0baf6486-41d8-4f73-8ce3-fde381e3d2ff" containerName="main"
Jul 27 17:46:16.041636 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.041611    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.044517 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.044463    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"kube-root-ca.crt\""
Jul 27 17:46:16.044650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.044525    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"default-dockercfg-zh7c5\""
Jul 27 17:46:16.044650 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.044628    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"openshift-service-ca.crt\""
Jul 27 17:46:16.045674 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.045658    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 27 17:46:16.056293 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.056267    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"]
Jul 27 17:46:16.143434 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.143401    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.143654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.143442    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22kmj\" (UniqueName: \"kubernetes.io/projected/c91913fd-4a7e-4848-9c1c-28950caab3bb-kube-api-access-22kmj\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.143654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.143555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.143654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.143587    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.143654 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.143624    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.143813 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.143674    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91913fd-4a7e-4848-9c1c-28950caab3bb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244154 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244123    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244334 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244160    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244334 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244334 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91913fd-4a7e-4848-9c1c-28950caab3bb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244334 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244295    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22kmj\" (UniqueName: \"kubernetes.io/projected/c91913fd-4a7e-4848-9c1c-28950caab3bb-kube-api-access-22kmj\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244661 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244589    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.244899 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244875    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.245161 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.244895    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.246895 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.246866    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c91913fd-4a7e-4848-9c1c-28950caab3bb-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.247007 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.246943    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c91913fd-4a7e-4848-9c1c-28950caab3bb-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.253731 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.253698    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"]
Jul 27 17:46:16.257178 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.257158    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.259034 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.259006    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22kmj\" (UniqueName: \"kubernetes.io/projected/c91913fd-4a7e-4848-9c1c-28950caab3bb-kube-api-access-22kmj\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb\" (UID: \"c91913fd-4a7e-4848-9c1c-28950caab3bb\") " pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:46:16.259918 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.259899    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-48639af5\"/\"llmisvc-model-pvc-router-manage-e8706282-epp-sa-dockercfg-jg58z\""
Jul 27 17:46:16.267493 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.267451    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"]
Jul 27 17:46:16.345084 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.345003    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4hv7\" (UniqueName: \"kubernetes.io/projected/a5aafbfc-97c9-416a-89a1-97d6ca461fb0-kube-api-access-z4hv7\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7\" (UID: \"a5aafbfc-97c9-416a-89a1-97d6ca461fb0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.345084 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.345063    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aafbfc-97c9-416a-89a1-97d6ca461fb0-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7\" (UID: \"a5aafbfc-97c9-416a-89a1-97d6ca461fb0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.445746 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.445716    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4hv7\" (UniqueName: \"kubernetes.io/projected/a5aafbfc-97c9-416a-89a1-97d6ca461fb0-kube-api-access-z4hv7\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7\" (UID: \"a5aafbfc-97c9-416a-89a1-97d6ca461fb0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.445934 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.445766    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aafbfc-97c9-416a-89a1-97d6ca461fb0-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7\" (UID: \"a5aafbfc-97c9-416a-89a1-97d6ca461fb0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.448424 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.448393    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5aafbfc-97c9-416a-89a1-97d6ca461fb0-tls-certs\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7\" (UID: \"a5aafbfc-97c9-416a-89a1-97d6ca461fb0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.453680 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.453657    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4hv7\" (UniqueName: \"kubernetes.io/projected/a5aafbfc-97c9-416a-89a1-97d6ca461fb0-kube-api-access-z4hv7\") pod \"llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7\" (UID: \"a5aafbfc-97c9-416a-89a1-97d6ca461fb0\") " pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.579394 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.579359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:16.717378 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:16.717347    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"]
Jul 27 17:46:16.720800 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:46:16.720772    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5aafbfc_97c9_416a_89a1_97d6ca461fb0.slice/crio-04d8c2e996d50661ee453fc28124ad20fae80e3318e8328b1280bee4ef10e52e WatchSource:0}: Error finding container 04d8c2e996d50661ee453fc28124ad20fae80e3318e8328b1280bee4ef10e52e: Status 404 returned error can't find the container with id 04d8c2e996d50661ee453fc28124ad20fae80e3318e8328b1280bee4ef10e52e
Jul 27 17:46:17.521354 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:17.521315    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7" event={"ID":"a5aafbfc-97c9-416a-89a1-97d6ca461fb0","Type":"ContainerStarted","Data":"3aaec604d8257b80c2eae719f99703b2b75d15ecbc774fb91b385b0f15bff37b"}
Jul 27 17:46:17.521820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:17.521379    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7" event={"ID":"a5aafbfc-97c9-416a-89a1-97d6ca461fb0","Type":"ContainerStarted","Data":"04d8c2e996d50661ee453fc28124ad20fae80e3318e8328b1280bee4ef10e52e"}
Jul 27 17:46:17.521820 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:17.521430    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:46:17.545932 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:17.545876    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7" podStartSLOduration=1.545860529 podStartE2EDuration="1.545860529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:46:17.545034336 +0000 UTC m=+2686.130395885" watchObservedRunningTime="2026-07-27 17:46:17.545860529 +0000 UTC m=+2686.131222086"
Jul 27 17:46:31.934892 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:31.934817    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:46:31.953063 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:31.953039    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:46:48.527355 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:46:48.527322    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-48639af5/llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7"
Jul 27 17:48:19.052704 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:48:19.052662    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 17:48:20.040994 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:48:20.040963    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:50:23.041749 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:50:23.041700    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 17:50:23.579758 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:50:23.579729    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:51:31.958085 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:51:31.958055    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:51:31.978106 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:51:31.978082    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:52:26.580366 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:52:26.580269    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 17:52:27.100343 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:52:27.100308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:54:30.101890 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:54:30.101829    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 17:54:30.659370 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:54:30.659338    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:56:30.000432 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.000395    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"]
Jul 27 17:56:30.009874 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:56:30.009839    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 17:56:30.009874 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:56:30.009874 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:56:30.009874 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d" cmd=["/bin/sleep","15"]
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.009896    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" message="" err=<
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:56:30.009908    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerName="main" containerID="cri-o://791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"
Jul 27 17:56:30.010173 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.009954    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerName="main" containerID="cri-o://791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d" gracePeriod=60
Jul 27 17:56:30.364286 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.364256    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:56:30.445027 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.444992    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tls-certs\") pod \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") "
Jul 27 17:56:30.445226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445042    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tmp-dir\") pod \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") "
Jul 27 17:56:30.445226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445105    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr77\" (UniqueName: \"kubernetes.io/projected/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-kube-api-access-vsr77\") pod \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") "
Jul 27 17:56:30.445226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445137    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-dshm\") pod \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") "
Jul 27 17:56:30.445226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445160    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-home\") pod \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") "
Jul 27 17:56:30.445226 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445182    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-model-cache\") pod \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\" (UID: \"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3\") "
Jul 27 17:56:30.445516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445369    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" (UID: "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:30.445516 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445445    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-home" (OuterVolumeSpecName: "home") pod "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" (UID: "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:30.445642 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.445604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" (UID: "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:30.447513 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.447462    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-dshm" (OuterVolumeSpecName: "dshm") pod "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" (UID: "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 17:56:30.447795 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.447777    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" (UID: "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 17:56:30.447874 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.447817    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-kube-api-access-vsr77" (OuterVolumeSpecName: "kube-api-access-vsr77") pod "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" (UID: "d1fb8aee-0cd9-48b3-bd26-b59358ff12f3"). InnerVolumeSpecName "kube-api-access-vsr77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 17:56:30.546534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.546434    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:56:30.546534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.546461    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:56:30.546534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.546489    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr77\" (UniqueName: \"kubernetes.io/projected/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-kube-api-access-vsr77\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:56:30.546534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.546500    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:56:30.546534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.546509    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:56:30.546534 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:30.546516    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 17:56:31.189972 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.189932    2579 generic.go:356] "Generic (PLEG): container finished" podID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerID="791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d" exitCode=0
Jul 27 17:56:31.190509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.189995    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"
Jul 27 17:56:31.190509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.189991    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" event={"ID":"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3","Type":"ContainerDied","Data":"791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"}
Jul 27 17:56:31.190509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.190098    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" event={"ID":"d1fb8aee-0cd9-48b3-bd26-b59358ff12f3","Type":"ContainerDied","Data":"03551959aa0b43b29d1c7390eb5b20a2bccdfdf1f83c1528c2ef24190f858945"}
Jul 27 17:56:31.190509 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.190139    2579 scope.go:117] "RemoveContainer" containerID="791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"
Jul 27 17:56:31.198881 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.198863    2579 scope.go:117] "RemoveContainer" containerID="791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"
Jul 27 17:56:31.199124 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:56:31.199105    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d\": container with ID starting with 791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d not found: ID does not exist" containerID="791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"
Jul 27 17:56:31.199186 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.199137    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d"} err="failed to get container status \"791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d\": rpc error: code = NotFound desc = could not find container \"791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d\": container with ID starting with 791957c0c8596116b71c348503c9cf3758d21c44b04d735e6e357126c953572d not found: ID does not exist"
Jul 27 17:56:31.213002 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.212968    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"]
Jul 27 17:56:31.217391 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.217367    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95"]
Jul 27 17:56:31.349125 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.349082    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-autoscaling-keda-deployment-b2150d0b/autoscale-keda-deploy-kserve-77fc6d7cf9-hdf95" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 27 17:56:31.892462 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.892424    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" path="/var/lib/kubelet/pods/d1fb8aee-0cd9-48b3-bd26-b59358ff12f3/volumes"
Jul 27 17:56:31.981257 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:31.981230    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:56:32.001959 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:32.001938    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 17:56:33.660904 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:56:33.660860    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 17:56:34.204306 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:56:34.204262    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 17:57:20.530015 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.529983    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 27 17:57:20.530551 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.530329    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerName="main"
Jul 27 17:57:20.530551 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.530342    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerName="main"
Jul 27 17:57:20.530551 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.530410    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1fb8aee-0cd9-48b3-bd26-b59358ff12f3" containerName="main"
Jul 27 17:57:20.533248 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.533233    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.536850 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.536826    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-mn-dockercfg-97tmf\""
Jul 27 17:57:20.536850 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.536841    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"kube-root-ca.crt\""
Jul 27 17:57:20.537128 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.537104    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"openshift-service-ca.crt\""
Jul 27 17:57:20.537232 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.537135    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"autoscale-hpa-lws-kserve-self-signed-certs\""
Jul 27 17:57:20.537232 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.537162    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2\"/\"default-dockercfg-8jmgg\""
Jul 27 17:57:20.545255 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.545233    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 27 17:57:20.661804 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.661770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.661996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.661828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.661996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.661887    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95ktt\" (UniqueName: \"kubernetes.io/projected/d311a425-f880-418b-b958-535d18a0da72-kube-api-access-95ktt\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.661996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.661950    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.661996 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.661987    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a425-f880-418b-b958-535d18a0da72-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.662182 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.662050    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763023 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.762985    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763220 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763035    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95ktt\" (UniqueName: \"kubernetes.io/projected/d311a425-f880-418b-b958-535d18a0da72-kube-api-access-95ktt\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763220 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763055    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763220 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763171    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a425-f880-418b-b958-535d18a0da72-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763236    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763398 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763517 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763404    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-tmp-dir\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763517 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763442    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-model-cache\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.763590 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.763551    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-home\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.765566 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.765539    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-dshm\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.765741 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.765723    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a425-f880-418b-b958-535d18a0da72-tls-certs\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.775593 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.775573    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95ktt\" (UniqueName: \"kubernetes.io/projected/d311a425-f880-418b-b958-535d18a0da72-kube-api-access-95ktt\") pod \"autoscale-hpa-lws-kserve-mn-0\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") " pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.842928 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.842837    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:20.987389 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:20.987363    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 27 17:57:20.989206 ip-10-0-132-194 kubenswrapper[2579]: W0727 17:57:20.989180    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd311a425_f880_418b_b958_535d18a0da72.slice/crio-717d84d71d3275e82ced598000dde7fd6c5920e89e09a4eef44257b2893d5c56 WatchSource:0}: Error finding container 717d84d71d3275e82ced598000dde7fd6c5920e89e09a4eef44257b2893d5c56: Status 404 returned error can't find the container with id 717d84d71d3275e82ced598000dde7fd6c5920e89e09a4eef44257b2893d5c56
Jul 27 17:57:21.404335 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:21.404297    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d311a425-f880-418b-b958-535d18a0da72","Type":"ContainerStarted","Data":"3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"}
Jul 27 17:57:21.404538 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:21.404357    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d311a425-f880-418b-b958-535d18a0da72","Type":"ContainerStarted","Data":"717d84d71d3275e82ced598000dde7fd6c5920e89e09a4eef44257b2893d5c56"}
Jul 27 17:57:21.421214 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:21.421141    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podStartSLOduration=1.421120499 podStartE2EDuration="1.421120499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 17:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 17:57:21.420849754 +0000 UTC m=+3350.006211312" watchObservedRunningTime="2026-07-27 17:57:21.421120499 +0000 UTC m=+3350.006482056"
Jul 27 17:57:21.843925 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:21.843886    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:30.844017 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:30.843981    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:30.857129 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:30.857103    2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:57:31.460665 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:57:31.460634    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 17:58:37.205163 ip-10-0-132-194 kubenswrapper[2579]: E0727 17:58:37.205074    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 17:58:37.733459 ip-10-0-132-194 kubenswrapper[2579]: I0727 17:58:37.733428    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:00:40.734511 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:00:40.734449    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:00:41.270490 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:00:41.270450    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:00:53.328427 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:00:53.328395    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-48639af5_llmisvcb19f98874e050eec8ca94d49676113f0-kserve-router-scheh87d7_a5aafbfc-97c9-416a-89a1-97d6ca461fb0/main/0.log"
Jul 27 18:01:32.004152 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:01:32.004125    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:01:32.027630 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:01:32.027604    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:02:44.271701 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:02:44.271657    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:02:44.815460 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:02:44.815429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:03:14.301937 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.301898    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 27 18:03:14.305193 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.305176    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.309012 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.308986    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 27 18:03:14.309182 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.309119    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"kube-root-ca.crt\""
Jul 27 18:03:14.309182 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.309161    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"default-dockercfg-4r6xp\""
Jul 27 18:03:14.309305 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.309213    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"openshift-service-ca.crt\""
Jul 27 18:03:14.309362 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.309349    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7c3633d1\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-85z6x\""
Jul 27 18:03:14.315353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.315327    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 27 18:03:14.393243 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.393207    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.393420 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.393251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.393420 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.393298    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.393420 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.393389    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.393420 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.393419    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439d279f-61d1-4c5c-987f-24b405e3e83a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.494537 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494502    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.494722 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.494722 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494573    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.494722 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.494722 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494651    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439d279f-61d1-4c5c-987f-24b405e3e83a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.494722 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbxb4\" (UniqueName: \"kubernetes.io/projected/439d279f-61d1-4c5c-987f-24b405e3e83a-kube-api-access-rbxb4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.495000 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494842    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.495000 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494839    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.495000 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.494963    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.496966 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.496947    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.497223 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.497207    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439d279f-61d1-4c5c-987f-24b405e3e83a-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.596185 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.596097    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbxb4\" (UniqueName: \"kubernetes.io/projected/439d279f-61d1-4c5c-987f-24b405e3e83a-kube-api-access-rbxb4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:03:14.606323 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:03:14.606293    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbxb4\" (UniqueName: \"kubernetes.io/projected/439d279f-61d1-4c5c-987f-24b405e3e83a-kube-api-access-rbxb4\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") " pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:04:47.815952 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:04:47.815905    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:04:48.337699 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:04:48.337668    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:05:17.318936 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:05:17.318897    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="439d279f-61d1-4c5c-987f-24b405e3e83a"
Jul 27 18:05:17.458114 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:17.458081    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:05:55.232992 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.232903    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 27 18:05:55.235273 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:05:55.233172    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context canceled" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="439d279f-61d1-4c5c-987f-24b405e3e83a"
Jul 27 18:05:55.622283 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.622203    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:05:55.628980 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.628955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:05:55.786164 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786129    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-model-cache\") pod \"439d279f-61d1-4c5c-987f-24b405e3e83a\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") "
Jul 27 18:05:55.786353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786173    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-home\") pod \"439d279f-61d1-4c5c-987f-24b405e3e83a\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") "
Jul 27 18:05:55.786353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786213    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-tmp-dir\") pod \"439d279f-61d1-4c5c-987f-24b405e3e83a\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") "
Jul 27 18:05:55.786353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786229    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439d279f-61d1-4c5c-987f-24b405e3e83a-tls-certs\") pod \"439d279f-61d1-4c5c-987f-24b405e3e83a\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") "
Jul 27 18:05:55.786353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786248    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-dshm\") pod \"439d279f-61d1-4c5c-987f-24b405e3e83a\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") "
Jul 27 18:05:55.786353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786306    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbxb4\" (UniqueName: \"kubernetes.io/projected/439d279f-61d1-4c5c-987f-24b405e3e83a-kube-api-access-rbxb4\") pod \"439d279f-61d1-4c5c-987f-24b405e3e83a\" (UID: \"439d279f-61d1-4c5c-987f-24b405e3e83a\") "
Jul 27 18:05:55.786663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786423    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-model-cache" (OuterVolumeSpecName: "model-cache") pod "439d279f-61d1-4c5c-987f-24b405e3e83a" (UID: "439d279f-61d1-4c5c-987f-24b405e3e83a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:55.786663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786465    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "439d279f-61d1-4c5c-987f-24b405e3e83a" (UID: "439d279f-61d1-4c5c-987f-24b405e3e83a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:55.786663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786501    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-home" (OuterVolumeSpecName: "home") pod "439d279f-61d1-4c5c-987f-24b405e3e83a" (UID: "439d279f-61d1-4c5c-987f-24b405e3e83a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:55.786663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786598    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:05:55.786663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786610    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:05:55.786663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.786625    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:05:55.788634 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.788604    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-dshm" (OuterVolumeSpecName: "dshm") pod "439d279f-61d1-4c5c-987f-24b405e3e83a" (UID: "439d279f-61d1-4c5c-987f-24b405e3e83a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:05:55.788634 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.788618    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/439d279f-61d1-4c5c-987f-24b405e3e83a-kube-api-access-rbxb4" (OuterVolumeSpecName: "kube-api-access-rbxb4") pod "439d279f-61d1-4c5c-987f-24b405e3e83a" (UID: "439d279f-61d1-4c5c-987f-24b405e3e83a"). InnerVolumeSpecName "kube-api-access-rbxb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:05:55.788780 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.788687    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/439d279f-61d1-4c5c-987f-24b405e3e83a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "439d279f-61d1-4c5c-987f-24b405e3e83a" (UID: "439d279f-61d1-4c5c-987f-24b405e3e83a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:05:55.887262 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.887180    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbxb4\" (UniqueName: \"kubernetes.io/projected/439d279f-61d1-4c5c-987f-24b405e3e83a-kube-api-access-rbxb4\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:05:55.887262 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.887211    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/439d279f-61d1-4c5c-987f-24b405e3e83a-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:05:55.887262 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:55.887225    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/439d279f-61d1-4c5c-987f-24b405e3e83a-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:05:56.626707 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:56.626674    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 27 18:05:56.660339 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:56.660310    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 27 18:05:56.665066 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:56.665043    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7c3633d1/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 27 18:05:57.892807 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:05:57.892772    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="439d279f-61d1-4c5c-987f-24b405e3e83a" path="/var/lib/kubelet/pods/439d279f-61d1-4c5c-987f-24b405e3e83a/volumes"
Jul 27 18:06:32.025869 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:06:32.025839    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:06:32.052609 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:06:32.052581    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:06:51.339151 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:06:51.339107    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:06:51.856671 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:06:51.856640    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:07:53.052157 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.052077    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"]
Jul 27 18:07:53.055239 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.055221    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.059189 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.059150    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 27 18:07:53.059353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.059236    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 27 18:07:53.059353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.059238    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 27 18:07:53.059353 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.059322    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-npjbj\""
Jul 27 18:07:53.059530 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.059384    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-htc6m\""
Jul 27 18:07:53.067266 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.067240    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"]
Jul 27 18:07:53.194794 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.194753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6ac7c2-cdce-469d-845b-77ea694e3987-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.194979 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.194828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx54z\" (UniqueName: \"kubernetes.io/projected/1a6ac7c2-cdce-469d-845b-77ea694e3987-kube-api-access-jx54z\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.296325 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.296285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6ac7c2-cdce-469d-845b-77ea694e3987-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.296576 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.296361    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx54z\" (UniqueName: \"kubernetes.io/projected/1a6ac7c2-cdce-469d-845b-77ea694e3987-kube-api-access-jx54z\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.299094 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.299069    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6ac7c2-cdce-469d-845b-77ea694e3987-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.305867 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.305810    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx54z\" (UniqueName: \"kubernetes.io/projected/1a6ac7c2-cdce-469d-845b-77ea694e3987-kube-api-access-jx54z\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.366403 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.366349    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:53.502339 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:53.502297    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"]
Jul 27 18:07:53.505669 ip-10-0-132-194 kubenswrapper[2579]: W0727 18:07:53.505641    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6ac7c2_cdce_469d_845b_77ea694e3987.slice/crio-ee934c368bfd7eafd06bc78293a6def0a2bbc207db7c272b0f5ba8824deec11a WatchSource:0}: Error finding container ee934c368bfd7eafd06bc78293a6def0a2bbc207db7c272b0f5ba8824deec11a: Status 404 returned error can't find the container with id ee934c368bfd7eafd06bc78293a6def0a2bbc207db7c272b0f5ba8824deec11a
Jul 27 18:07:54.125861 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:54.125821    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" event={"ID":"1a6ac7c2-cdce-469d-845b-77ea694e3987","Type":"ContainerStarted","Data":"7e290a6a819c988058e668af6b70e97e5427c7e62bbd4ad418915cab8e001eec"}
Jul 27 18:07:54.126286 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:54.125882    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" event={"ID":"1a6ac7c2-cdce-469d-845b-77ea694e3987","Type":"ContainerStarted","Data":"ee934c368bfd7eafd06bc78293a6def0a2bbc207db7c272b0f5ba8824deec11a"}
Jul 27 18:07:54.126286 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:54.125963    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:07:54.149918 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:07:54.149857    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" podStartSLOduration=1.149842304 podStartE2EDuration="1.149842304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:07:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:07:54.148058876 +0000 UTC m=+3982.733420446" watchObservedRunningTime="2026-07-27 18:07:54.149842304 +0000 UTC m=+3982.735203860"
Jul 27 18:08:25.133051 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:08:25.133022    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:08:54.858191 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:08:54.858088    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:08:55.394152 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:08:55.394117    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:10:21.568394 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:21.568352    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"]
Jul 27 18:10:25.132619 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:25.132537    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:10:35.133095 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:35.133059    2579 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Jul 27 18:10:36.587929 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.587877    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerName="main" containerID="cri-o://7e290a6a819c988058e668af6b70e97e5427c7e62bbd4ad418915cab8e001eec" gracePeriod=45
Jul 27 18:10:36.836071 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.836039    2579 generic.go:356] "Generic (PLEG): container finished" podID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerID="7e290a6a819c988058e668af6b70e97e5427c7e62bbd4ad418915cab8e001eec" exitCode=0
Jul 27 18:10:36.836211 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.836114    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" event={"ID":"1a6ac7c2-cdce-469d-845b-77ea694e3987","Type":"ContainerDied","Data":"7e290a6a819c988058e668af6b70e97e5427c7e62bbd4ad418915cab8e001eec"}
Jul 27 18:10:36.836211 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.836207    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4" event={"ID":"1a6ac7c2-cdce-469d-845b-77ea694e3987","Type":"ContainerDied","Data":"ee934c368bfd7eafd06bc78293a6def0a2bbc207db7c272b0f5ba8824deec11a"}
Jul 27 18:10:36.836292 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.836233    2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee934c368bfd7eafd06bc78293a6def0a2bbc207db7c272b0f5ba8824deec11a"
Jul 27 18:10:36.840312 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.840255    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:10:36.910820 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.910780    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jx54z\" (UniqueName: \"kubernetes.io/projected/1a6ac7c2-cdce-469d-845b-77ea694e3987-kube-api-access-jx54z\") pod \"1a6ac7c2-cdce-469d-845b-77ea694e3987\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") "
Jul 27 18:10:36.911013 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.910863    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6ac7c2-cdce-469d-845b-77ea694e3987-tls-certs\") pod \"1a6ac7c2-cdce-469d-845b-77ea694e3987\" (UID: \"1a6ac7c2-cdce-469d-845b-77ea694e3987\") "
Jul 27 18:10:36.913161 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.913133    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6ac7c2-cdce-469d-845b-77ea694e3987-kube-api-access-jx54z" (OuterVolumeSpecName: "kube-api-access-jx54z") pod "1a6ac7c2-cdce-469d-845b-77ea694e3987" (UID: "1a6ac7c2-cdce-469d-845b-77ea694e3987"). InnerVolumeSpecName "kube-api-access-jx54z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:10:36.913278 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:36.913206    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6ac7c2-cdce-469d-845b-77ea694e3987-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a6ac7c2-cdce-469d-845b-77ea694e3987" (UID: "1a6ac7c2-cdce-469d-845b-77ea694e3987"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:10:37.011466 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:37.011433    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx54z\" (UniqueName: \"kubernetes.io/projected/1a6ac7c2-cdce-469d-845b-77ea694e3987-kube-api-access-jx54z\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.011466 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:37.011461    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a6ac7c2-cdce-469d-845b-77ea694e3987-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:10:37.840501 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:37.840447    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"
Jul 27 18:10:37.865340 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:37.865312    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"]
Jul 27 18:10:37.871522 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:37.871495    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-bxcl4"]
Jul 27 18:10:37.892819 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:37.892787    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" path="/var/lib/kubelet/pods/1a6ac7c2-cdce-469d-845b-77ea694e3987/volumes"
Jul 27 18:10:39.761011 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.760976    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"]
Jul 27 18:10:39.761487 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.761453    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerName="main"
Jul 27 18:10:39.761550 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.761506    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerName="main"
Jul 27 18:10:39.761615 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.761603    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a6ac7c2-cdce-469d-845b-77ea694e3987" containerName="main"
Jul 27 18:10:39.764915 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.764897    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:39.768550 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.768530    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Jul 27 18:10:39.769364 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.769345    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-9xch9\""
Jul 27 18:10:39.769667 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.769644    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-npjbj\""
Jul 27 18:10:39.769784 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.769669    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 27 18:10:39.769784 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.769689    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Jul 27 18:10:39.776171 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.776150    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"]
Jul 27 18:10:39.837591 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.837556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn6gh\" (UniqueName: \"kubernetes.io/projected/5bedd631-9f5a-4fbd-b885-b9a24d7307ce-kube-api-access-jn6gh\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29\" (UID: \"5bedd631-9f5a-4fbd-b885-b9a24d7307ce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:39.837757 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.837612    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd631-9f5a-4fbd-b885-b9a24d7307ce-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29\" (UID: \"5bedd631-9f5a-4fbd-b885-b9a24d7307ce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:39.938517 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.938460    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn6gh\" (UniqueName: \"kubernetes.io/projected/5bedd631-9f5a-4fbd-b885-b9a24d7307ce-kube-api-access-jn6gh\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29\" (UID: \"5bedd631-9f5a-4fbd-b885-b9a24d7307ce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:39.938676 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.938553    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd631-9f5a-4fbd-b885-b9a24d7307ce-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29\" (UID: \"5bedd631-9f5a-4fbd-b885-b9a24d7307ce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:39.941088 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.941070    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bedd631-9f5a-4fbd-b885-b9a24d7307ce-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29\" (UID: \"5bedd631-9f5a-4fbd-b885-b9a24d7307ce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:39.948329 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:39.948304    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn6gh\" (UniqueName: \"kubernetes.io/projected/5bedd631-9f5a-4fbd-b885-b9a24d7307ce-kube-api-access-jn6gh\") pod \"stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29\" (UID: \"5bedd631-9f5a-4fbd-b885-b9a24d7307ce\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:40.076172 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:40.076089    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:40.212834 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:40.212794    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"]
Jul 27 18:10:40.215568 ip-10-0-132-194 kubenswrapper[2579]: W0727 18:10:40.215541    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bedd631_9f5a_4fbd_b885_b9a24d7307ce.slice/crio-2b0740b39fb408247391f9afc47736dc3e9dd4fb617aadec0556f67dd7892e8e WatchSource:0}: Error finding container 2b0740b39fb408247391f9afc47736dc3e9dd4fb617aadec0556f67dd7892e8e: Status 404 returned error can't find the container with id 2b0740b39fb408247391f9afc47736dc3e9dd4fb617aadec0556f67dd7892e8e
Jul 27 18:10:40.857095 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:40.857057    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29" event={"ID":"5bedd631-9f5a-4fbd-b885-b9a24d7307ce","Type":"ContainerStarted","Data":"5364c01ee1437729dd885bc22cdd8d245630eaab6f53e56352b29b959fa11138"}
Jul 27 18:10:40.857519 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:40.857123    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:10:40.857519 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:40.857147    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29" event={"ID":"5bedd631-9f5a-4fbd-b885-b9a24d7307ce","Type":"ContainerStarted","Data":"2b0740b39fb408247391f9afc47736dc3e9dd4fb617aadec0556f67dd7892e8e"}
Jul 27 18:10:40.877938 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:40.877887    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29" podStartSLOduration=1.8778722989999999 podStartE2EDuration="1.877872299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:10:40.87656679 +0000 UTC m=+4149.461928349" watchObservedRunningTime="2026-07-27 18:10:40.877872299 +0000 UTC m=+4149.463233883"
Jul 27 18:10:58.395411 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:10:58.395365    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:10:58.932630 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:10:58.932599    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:11:11.863531 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:11:11.863500    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-6bbdff5b46-psh29"
Jul 27 18:11:32.052017 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:11:32.051991    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:11:32.076620 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:11:32.076589    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:12:09.241003 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:12:09.240965    2579 generic.go:356] "Generic (PLEG): container finished" podID="73f0dfdd-b806-4dcb-9897-65cf2e003c69" containerID="f556575041d5923e4fd3135266258f8100f942d3e020e6f558b28ff1ca648c53" exitCode=1
Jul 27 18:12:09.241580 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:12:09.241044    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8" event={"ID":"73f0dfdd-b806-4dcb-9897-65cf2e003c69","Type":"ContainerDied","Data":"f556575041d5923e4fd3135266258f8100f942d3e020e6f558b28ff1ca648c53"}
Jul 27 18:12:09.241580 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:12:09.241524    2579 scope.go:117] "RemoveContainer" containerID="f556575041d5923e4fd3135266258f8100f942d3e020e6f558b28ff1ca648c53"
Jul 27 18:12:10.248095 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:12:10.248059    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8" event={"ID":"73f0dfdd-b806-4dcb-9897-65cf2e003c69","Type":"ContainerStarted","Data":"baacca3a562be3fe4cd69de65be76e6dd4b77457ee09984c357e276aa9a87f23"}
Jul 27 18:12:10.248555 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:12:10.248261    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 18:12:21.254706 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:12:21.254674    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-56fb6f6856-n5pb8"
Jul 27 18:13:01.933694 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:13:01.933647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:13:02.487973 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:02.487942    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:13:32.915811 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:32.915730    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 27 18:13:32.925018 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:13:32.924978    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:13:32.925018 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:13:32.925018 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 18:13:32.925018 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c" cmd=["/bin/sleep","15"]
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:32.925039    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" message="" err=<
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:13:32.925055    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="d311a425-f880-418b-b958-535d18a0da72" containerName="main" containerID="cri-o://3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"
Jul 27 18:13:32.925275 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:32.925103    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" podUID="d311a425-f880-418b-b958-535d18a0da72" containerName="main" containerID="cri-o://3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c" gracePeriod=60
Jul 27 18:13:33.276286 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.276261    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 18:13:33.439936 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.439840    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-model-cache\") pod \"d311a425-f880-418b-b958-535d18a0da72\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") "
Jul 27 18:13:33.439936 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.439926    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-dshm\") pod \"d311a425-f880-418b-b958-535d18a0da72\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") "
Jul 27 18:13:33.440183 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.439948    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95ktt\" (UniqueName: \"kubernetes.io/projected/d311a425-f880-418b-b958-535d18a0da72-kube-api-access-95ktt\") pod \"d311a425-f880-418b-b958-535d18a0da72\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") "
Jul 27 18:13:33.440183 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.439971    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-home\") pod \"d311a425-f880-418b-b958-535d18a0da72\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") "
Jul 27 18:13:33.440183 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.439991    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a425-f880-418b-b958-535d18a0da72-tls-certs\") pod \"d311a425-f880-418b-b958-535d18a0da72\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") "
Jul 27 18:13:33.440183 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.440016    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-tmp-dir\") pod \"d311a425-f880-418b-b958-535d18a0da72\" (UID: \"d311a425-f880-418b-b958-535d18a0da72\") "
Jul 27 18:13:33.440376 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.440189    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-model-cache" (OuterVolumeSpecName: "model-cache") pod "d311a425-f880-418b-b958-535d18a0da72" (UID: "d311a425-f880-418b-b958-535d18a0da72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:13:33.440376 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.440211    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-home" (OuterVolumeSpecName: "home") pod "d311a425-f880-418b-b958-535d18a0da72" (UID: "d311a425-f880-418b-b958-535d18a0da72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:13:33.440376 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.440337    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:13:33.440376 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.440356    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:13:33.440580 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.440466    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d311a425-f880-418b-b958-535d18a0da72" (UID: "d311a425-f880-418b-b958-535d18a0da72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:13:33.442489 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.442449    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d311a425-f880-418b-b958-535d18a0da72-kube-api-access-95ktt" (OuterVolumeSpecName: "kube-api-access-95ktt") pod "d311a425-f880-418b-b958-535d18a0da72" (UID: "d311a425-f880-418b-b958-535d18a0da72"). InnerVolumeSpecName "kube-api-access-95ktt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:13:33.442608 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.442448    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d311a425-f880-418b-b958-535d18a0da72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d311a425-f880-418b-b958-535d18a0da72" (UID: "d311a425-f880-418b-b958-535d18a0da72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:13:33.442608 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.442520    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-dshm" (OuterVolumeSpecName: "dshm") pod "d311a425-f880-418b-b958-535d18a0da72" (UID: "d311a425-f880-418b-b958-535d18a0da72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:13:33.541176 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.541135    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:13:33.541176 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.541167    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d311a425-f880-418b-b958-535d18a0da72-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:13:33.541176 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.541181    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95ktt\" (UniqueName: \"kubernetes.io/projected/d311a425-f880-418b-b958-535d18a0da72-kube-api-access-95ktt\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:13:33.541424 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.541193    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d311a425-f880-418b-b958-535d18a0da72-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:13:33.622969 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.622934    2579 generic.go:356] "Generic (PLEG): container finished" podID="d311a425-f880-418b-b958-535d18a0da72" containerID="3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c" exitCode=0
Jul 27 18:13:33.623162 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.622980    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"
Jul 27 18:13:33.623162 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.623009    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d311a425-f880-418b-b958-535d18a0da72","Type":"ContainerDied","Data":"3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"}
Jul 27 18:13:33.623162 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.623065    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0" event={"ID":"d311a425-f880-418b-b958-535d18a0da72","Type":"ContainerDied","Data":"717d84d71d3275e82ced598000dde7fd6c5920e89e09a4eef44257b2893d5c56"}
Jul 27 18:13:33.623162 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.623093    2579 scope.go:117] "RemoveContainer" containerID="3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"
Jul 27 18:13:33.632673 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.632655    2579 scope.go:117] "RemoveContainer" containerID="3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"
Jul 27 18:13:33.632938 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:13:33.632920    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c\": container with ID starting with 3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c not found: ID does not exist" containerID="3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"
Jul 27 18:13:33.633007 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.632947    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c"} err="failed to get container status \"3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c\": rpc error: code = NotFound desc = could not find container \"3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c\": container with ID starting with 3745fe923b66122d90c411366f42b9f3f5f7a53f6cd3ff6d42a9dd425f574f4c not found: ID does not exist"
Jul 27 18:13:33.651307 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.651274    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 27 18:13:33.750570 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.750536    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-hpa-lws-d4cbcfd2/autoscale-hpa-lws-kserve-mn-0"]
Jul 27 18:13:33.893167 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:13:33.893132    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d311a425-f880-418b-b958-535d18a0da72" path="/var/lib/kubelet/pods/d311a425-f880-418b-b958-535d18a0da72/volumes"
Jul 27 18:14:06.810263 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.810224    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:14:06.810814 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.810566    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d311a425-f880-418b-b958-535d18a0da72" containerName="main"
Jul 27 18:14:06.810814 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.810578    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d311a425-f880-418b-b958-535d18a0da72" containerName="main"
Jul 27 18:14:06.810814 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.810632    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d311a425-f880-418b-b958-535d18a0da72" containerName="main"
Jul 27 18:14:06.813449 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.813424    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:06.816617 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.816593    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-mn-dockercfg-kdpwr\""
Jul 27 18:14:06.817755 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.817732    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"kube-root-ca.crt\""
Jul 27 18:14:06.817904 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.817809    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"openshift-service-ca.crt\""
Jul 27 18:14:06.817988 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.817922    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"default-dockercfg-7vr7h\""
Jul 27 18:14:06.817988 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.817954    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-autoscaling-keda-lws-e541a132\"/\"autoscale-keda-lws-kserve-self-signed-certs\""
Jul 27 18:14:06.827418 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.827393    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:14:06.933689 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.933655    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:06.933689 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.933688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:06.933911 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.933717    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:06.933911 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.933803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41e4139-040c-42b7-84cc-e15cf2105d54-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:06.933911 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.933863    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:06.933911 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:06.933889    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rjz\" (UniqueName: \"kubernetes.io/projected/e41e4139-040c-42b7-84cc-e15cf2105d54-kube-api-access-86rjz\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035116 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41e4139-040c-42b7-84cc-e15cf2105d54-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035288 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035288 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rjz\" (UniqueName: \"kubernetes.io/projected/e41e4139-040c-42b7-84cc-e15cf2105d54-kube-api-access-86rjz\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035288 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035288 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035243    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035506 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035649 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035622    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-model-cache\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035723 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035678    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-tmp-dir\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.035768 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.035745    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-home\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.037538 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.037518    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-dshm\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.037648 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.037637    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41e4139-040c-42b7-84cc-e15cf2105d54-tls-certs\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.048780 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.048756    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rjz\" (UniqueName: \"kubernetes.io/projected/e41e4139-040c-42b7-84cc-e15cf2105d54-kube-api-access-86rjz\") pod \"autoscale-keda-lws-kserve-mn-0-1\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") " pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.123818 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.123738    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:14:07.269501 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.269453    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:14:07.271582 ip-10-0-132-194 kubenswrapper[2579]: W0727 18:14:07.271542    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41e4139_040c_42b7_84cc_e15cf2105d54.slice/crio-5c89dbb1315e8cedc2aadeee990581abbb3191505c7cdfceafe57f8474213d0f WatchSource:0}: Error finding container 5c89dbb1315e8cedc2aadeee990581abbb3191505c7cdfceafe57f8474213d0f: Status 404 returned error can't find the container with id 5c89dbb1315e8cedc2aadeee990581abbb3191505c7cdfceafe57f8474213d0f
Jul 27 18:14:07.783762 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.783718    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"e41e4139-040c-42b7-84cc-e15cf2105d54","Type":"ContainerStarted","Data":"301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"}
Jul 27 18:14:07.783926 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.783775    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"e41e4139-040c-42b7-84cc-e15cf2105d54","Type":"ContainerStarted","Data":"5c89dbb1315e8cedc2aadeee990581abbb3191505c7cdfceafe57f8474213d0f"}
Jul 27 18:14:07.801688 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:07.801642    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podStartSLOduration=1.801626704 podStartE2EDuration="1.801626704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 18:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:14:07.799907024 +0000 UTC m=+4356.385268577" watchObservedRunningTime="2026-07-27 18:14:07.801626704 +0000 UTC m=+4356.386988340"
Jul 27 18:14:32.709102 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:14:32.709068    2579 scope.go:117] "RemoveContainer" containerID="7e290a6a819c988058e668af6b70e97e5427c7e62bbd4ad418915cab8e001eec"
Jul 27 18:15:05.488984 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:15:05.488926    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:15:06.043674 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:15:06.043634    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:16:32.075007 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:16:32.074933    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:16:32.103518 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:16:32.103495    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:17:09.044401 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:17:09.044358    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:17:09.573175 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:17:09.573137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:19:12.574191 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:19:12.574144    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:19:13.125107 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:19:13.125080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:21:16.125732 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:21:16.125688    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:21:16.665766 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:21:16.665733    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:21:32.098441 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:21:32.098338    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:21:32.127987 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:21:32.127966    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:23:19.666731 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:23:19.666686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:23:20.188626 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:23:20.188590    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:25:23.189907 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:25:23.189809    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:25:23.730596 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:25:23.730561    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:26:32.122043 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:26:32.121936    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:26:32.151394 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:26:32.151372    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:27:26.731259 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:27:26.731214    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:27:27.275343 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:27:27.275311    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:29:01.256258 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.256180    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:29:01.265419 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:29:01.265379    2579 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 27 18:29:01.265419 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:29:01.265419 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 18:29:01.265419 ip-10-0-132-194 kubenswrapper[2579]:  > containerID="301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b" cmd=["/bin/sleep","15"]
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.265429    2579 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" message="" err=<
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]:  >
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:29:01.265441    2579 kuberuntime_container.go:755] "PreStop hook failed" err=<
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]:         , stdout: , stderr: , exit code -1
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]:  > pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="e41e4139-040c-42b7-84cc-e15cf2105d54" containerName="main" containerID="cri-o://301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"
Jul 27 18:29:01.265756 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.265503    2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" podUID="e41e4139-040c-42b7-84cc-e15cf2105d54" containerName="main" containerID="cri-o://301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b" gracePeriod=60
Jul 27 18:29:01.518416 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.518349    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:29:01.619566 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619534    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-tmp-dir\") pod \"e41e4139-040c-42b7-84cc-e15cf2105d54\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") "
Jul 27 18:29:01.619748 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619576    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86rjz\" (UniqueName: \"kubernetes.io/projected/e41e4139-040c-42b7-84cc-e15cf2105d54-kube-api-access-86rjz\") pod \"e41e4139-040c-42b7-84cc-e15cf2105d54\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") "
Jul 27 18:29:01.619748 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619637    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-dshm\") pod \"e41e4139-040c-42b7-84cc-e15cf2105d54\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") "
Jul 27 18:29:01.619748 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619652    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41e4139-040c-42b7-84cc-e15cf2105d54-tls-certs\") pod \"e41e4139-040c-42b7-84cc-e15cf2105d54\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") "
Jul 27 18:29:01.619748 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619679    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-model-cache\") pod \"e41e4139-040c-42b7-84cc-e15cf2105d54\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") "
Jul 27 18:29:01.619963 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619790    2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-home\") pod \"e41e4139-040c-42b7-84cc-e15cf2105d54\" (UID: \"e41e4139-040c-42b7-84cc-e15cf2105d54\") "
Jul 27 18:29:01.619963 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619882    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e41e4139-040c-42b7-84cc-e15cf2105d54" (UID: "e41e4139-040c-42b7-84cc-e15cf2105d54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.619963 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.619919    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-model-cache" (OuterVolumeSpecName: "model-cache") pod "e41e4139-040c-42b7-84cc-e15cf2105d54" (UID: "e41e4139-040c-42b7-84cc-e15cf2105d54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.620118 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.620080    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-home" (OuterVolumeSpecName: "home") pod "e41e4139-040c-42b7-84cc-e15cf2105d54" (UID: "e41e4139-040c-42b7-84cc-e15cf2105d54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.620173 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.620160    2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-model-cache\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.620228 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.620178    2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-home\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.620228 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.620191    2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-tmp-dir\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.622044 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.622021    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-dshm" (OuterVolumeSpecName: "dshm") pod "e41e4139-040c-42b7-84cc-e15cf2105d54" (UID: "e41e4139-040c-42b7-84cc-e15cf2105d54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 27 18:29:01.622161 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.622040    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e41e4139-040c-42b7-84cc-e15cf2105d54-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e41e4139-040c-42b7-84cc-e15cf2105d54" (UID: "e41e4139-040c-42b7-84cc-e15cf2105d54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 27 18:29:01.622161 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.622050    2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e41e4139-040c-42b7-84cc-e15cf2105d54-kube-api-access-86rjz" (OuterVolumeSpecName: "kube-api-access-86rjz") pod "e41e4139-040c-42b7-84cc-e15cf2105d54" (UID: "e41e4139-040c-42b7-84cc-e15cf2105d54"). InnerVolumeSpecName "kube-api-access-86rjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 27 18:29:01.690683 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.690653    2579 generic.go:356] "Generic (PLEG): container finished" podID="e41e4139-040c-42b7-84cc-e15cf2105d54" containerID="301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b" exitCode=0
Jul 27 18:29:01.690848 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.690702    2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"
Jul 27 18:29:01.690848 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.690715    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"e41e4139-040c-42b7-84cc-e15cf2105d54","Type":"ContainerDied","Data":"301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"}
Jul 27 18:29:01.690848 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.690760    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1" event={"ID":"e41e4139-040c-42b7-84cc-e15cf2105d54","Type":"ContainerDied","Data":"5c89dbb1315e8cedc2aadeee990581abbb3191505c7cdfceafe57f8474213d0f"}
Jul 27 18:29:01.690848 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.690787    2579 scope.go:117] "RemoveContainer" containerID="301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"
Jul 27 18:29:01.700528 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.700391    2579 scope.go:117] "RemoveContainer" containerID="301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"
Jul 27 18:29:01.706921 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:29:01.706890    2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b\": container with ID starting with 301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b not found: ID does not exist" containerID="301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"
Jul 27 18:29:01.707113 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.707087    2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b"} err="failed to get container status \"301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b\": rpc error: code = NotFound desc = could not find container \"301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b\": container with ID starting with 301dee8688b538a3de936863f26283c02ce10446985195bb5c7e9bd18f1b737b not found: ID does not exist"
Jul 27 18:29:01.718034 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.718007    2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:29:01.720963 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.720932    2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rjz\" (UniqueName: \"kubernetes.io/projected/e41e4139-040c-42b7-84cc-e15cf2105d54-kube-api-access-86rjz\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.720963 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.720961    2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e41e4139-040c-42b7-84cc-e15cf2105d54-dshm\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.721094 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.720976    2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e41e4139-040c-42b7-84cc-e15cf2105d54-tls-certs\") on node \"ip-10-0-132-194.ec2.internal\" DevicePath \"\""
Jul 27 18:29:01.722292 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.722273    2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-autoscaling-keda-lws-e541a132/autoscale-keda-lws-kserve-mn-0-1"]
Jul 27 18:29:01.893253 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:01.893172    2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e41e4139-040c-42b7-84cc-e15cf2105d54" path="/var/lib/kubelet/pods/e41e4139-040c-42b7-84cc-e15cf2105d54/volumes"
Jul 27 18:29:30.276789 ip-10-0-132-194 kubenswrapper[2579]: E0727 18:29:30.276747    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kserve-pvc-source], unattached volumes=[kserve-pvc-source], failed to process volumes=[]: context deadline exceeded" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb" podUID="c91913fd-4a7e-4848-9c1c-28950caab3bb"
Jul 27 18:29:30.815816 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:30.815781    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-48639af5/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-7bqm6fb"
Jul 27 18:29:46.901756 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:46.901728    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:29:47.017465 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:47.017435    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:29:48.110324 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:48.110287    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:29:48.196612 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:48.196582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:29:49.331535 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:49.331505    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:29:49.452586 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:49.452553    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:29:50.515454 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:50.515421    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:29:50.647879 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:50.647850    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:29:51.602362 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:51.602331    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:51.605082 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:51.605054    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:51.898448 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:51.898337    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:51.908316 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:51.908293    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:51.926288 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:51.926264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:29:52.900632 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:52.900554    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:52.903314 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:52.903290    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:53.203938 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:53.203907    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:53.218072 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:53.218041    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:53.234827 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:53.234799    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:29:54.223551 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:54.223524    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:54.226558 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:54.226535    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:54.501783 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:54.501696    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:54.514618 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:54.514597    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:54.531370 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:54.531340    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:29:55.445767 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:55.445280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:55.448409 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:55.448365    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:55.735633 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:55.735577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:55.748340 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:55.748311    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:55.764966 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:55.764914    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:29:56.774026 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:56.773993    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:56.777494 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:56.777459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:57.059175 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:57.059092    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:57.074800 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:57.074771    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:57.091702 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:57.091672    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:29:58.006098 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:58.006063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:58.009053 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:58.009022    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:58.290115 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:58.290033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:58.303804 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:58.303778    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:58.320777 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:58.320750    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:29:59.246027 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:59.245999    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:29:59.249332 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:59.249305    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:29:59.538390 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:59.538299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:29:59.553954 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:59.553926    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:29:59.570741 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:29:59.570708    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:00.499342 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:00.499313    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:00.502296 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:00.502275    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:00.794771 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:00.794686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:00.813457 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:00.813406    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:00.830250 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:00.830224    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:01.774340 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:01.774308    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:01.777467 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:01.777445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:02.086091 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:02.086000    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:02.105123 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:02.105086    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:02.141929 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:02.141899    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:03.189984 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:03.189944    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:03.192936 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:03.192910    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:03.483044 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:03.483000    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:03.498161 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:03.498117    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:03.517764 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:03.517726    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:04.508899 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:04.508870    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:04.512039 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:04.512004    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:04.816846 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:04.816758    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:04.830268 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:04.830238    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:04.849684 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:04.849647    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:05.794063 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:05.794026    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:05.797036 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:05.797011    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:06.092285 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:06.092183    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:06.106288 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:06.106264    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:06.124063 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:06.124036    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:07.064754 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:07.064722    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:07.067341 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:07.067318    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:07.360233 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:07.360124    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:07.376487 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:07.376447    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:07.393670 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:07.393626    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:08.332067 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:08.332033    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:08.335266 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:08.335244    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:08.616545 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:08.616384    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:08.631498 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:08.631453    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:08.654842 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:08.654814    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:09.574376 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:09.574329    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:09.577128 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:09.577105    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:09.837047 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:09.836964    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:09.852441 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:09.852414    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:09.869042 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:09.869014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:10.770079 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:10.770045    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:10.774007 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:10.773982    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:11.056556 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:11.056450    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:11.072997 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:11.072972    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:11.090180 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:11.090157    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:12.005230 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:12.005203    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:12.008853 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:12.008829    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:12.280341 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:12.280256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:12.301033 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:12.301010    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:12.325142 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:12.325122    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:13.241931 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:13.241898    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:13.244669 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:13.244648    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:13.566942 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:13.566841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:13.587125 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:13.587092    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:13.610163 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:13.610137    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:14.537192 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:14.537162    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:14.539940 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:14.539903    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:14.824234 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:14.824143    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:14.839085 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:14.839057    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:14.856145 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:14.856125    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:15.784704 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:15.784674    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:15.787403 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:15.787380    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:16.070670 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:16.070579    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:16.087034 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:16.087002    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:16.103525 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:16.103501    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:17.031815 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:17.031782    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:17.035092 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:17.035066    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:17.274028 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:17.273992    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:17.289837 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:17.289762    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:17.306717 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:17.306692    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:18.243844 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:18.243811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:18.246892 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:18.246866    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:18.553701 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:18.553614    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:18.576955 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:18.576932    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:18.593570 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:18.593541    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:19.515609 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:19.515578    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:19.518504 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:19.518464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:19.810075 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:19.809979    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:19.825964 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:19.825934    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:19.843913 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:19.843884    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:20.783051 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:20.783018    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:20.785965 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:20.785946    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:21.072756 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:21.072670    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:21.087855 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:21.087830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:21.104723 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:21.104697    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:22.074211 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:22.074180    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:22.077183 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:22.077158    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:22.372145 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:22.372042    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:22.392332 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:22.392299    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:22.409257 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:22.409230    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:23.342689 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:23.342656    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:23.345713 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:23.345690    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:23.634896 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:23.634802    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:23.648904 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:23.648879    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:23.666944 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:23.666917    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:24.598736 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:24.598698    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:24.601851 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:24.601826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:24.895096 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:24.895009    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:24.913464 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:24.913433    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:24.932385 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:24.932355    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:25.863613 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:25.863582    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:25.866661 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:25.866638    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:26.148987 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:26.148895    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:26.167496 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:26.167458    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:26.191389 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:26.191359    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:27.132001 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:27.131954    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:27.135247 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:27.135212    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:27.417026 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:27.416935    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:27.430714 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:27.430686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:27.448725 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:27.448699    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:28.389218 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:28.389185    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:28.392250 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:28.392229    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:28.686984 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:28.686900    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:28.700255 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:28.700227    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:28.718002 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:28.717967    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:29.678814 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:29.678785    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:29.681823 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:29.681791    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:29.978727 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:29.978693    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:29.993840 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:29.993810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:30.010515 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:30.010491    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:30.966831 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:30.966797    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:30.969868 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:30.969841    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:31.272307 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:31.272205    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:31.288448 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:31.288410    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:31.306700 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:31.306673    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:32.264759 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:32.264725    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:32.267876 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:32.267852    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:32.558599 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:32.558508    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:32.572188 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:32.572155    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:32.588750 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:32.588717    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:33.558117 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:33.558089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:33.561085 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:33.561052    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:33.850709 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:33.850624    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:33.865453 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:33.865429    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:33.881663 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:33.881641    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:34.832782 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:34.832747    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:34.835541 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:34.835512    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:35.122982 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:35.122893    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:35.136813 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:35.136790    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:35.155169 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:35.155144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:36.118594 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:36.118563    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:36.122039 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:36.122009    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:36.413316 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:36.413227    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:36.432617 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:36.432584    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:36.451117 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:36.451089    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:37.423511 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:37.423459    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:37.426429 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:37.426407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:37.716102 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:37.716063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:37.733024 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:37.732998    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:37.750608 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:37.750577    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:38.700935 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:38.700902    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:38.704364 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:38.704340    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:38.990270 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:38.990237    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:39.007053 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:39.007009    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:39.024216 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:39.024189    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:39.948272 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:39.948241    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:39.951445 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:39.951422    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:40.241290 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:40.241256    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:40.256638 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:40.256609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:40.276115 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:40.276090    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:41.206539 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:41.206512    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:41.209685 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:41.209657    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:41.491941 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:41.491845    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:41.516061 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:41.516025    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:41.536287 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:41.536255    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:42.489235 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:42.489201    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:42.492064 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:42.492014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:42.787109 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:42.787014    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:42.803199 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:42.803163    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:42.819836 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:42.819808    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:43.760405 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:43.760368    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:43.763899 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:43.763871    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:44.052253 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:44.052176    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:44.069282 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:44.069255    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:44.087217 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:44.087193    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:45.044094 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:45.044067    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:45.047194 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:45.047169    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:45.340717 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:45.340619    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:45.358428 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:45.358398    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:45.375507 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:45.375464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:46.316379 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:46.316341    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:46.319595 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:46.319574    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:46.598829 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:46.598742    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:46.618841 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:46.618811    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:46.635695 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:46.635668    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:47.612311 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:47.612279    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:47.615761 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:47.615730    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:47.910658 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:47.910566    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:47.924451 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:47.924420    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:47.941819 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:47.941784    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:48.874553 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:48.874525    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:48.877495 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:48.877445    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:49.155436 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:49.155348    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:49.169498 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:49.169461    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:49.187165 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:49.187145    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:50.135449 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:50.135407    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:50.138116 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:50.138091    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:50.426765 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:50.426672    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:50.440041 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:50.440017    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:50.457742 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:50.457702    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:51.386314 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:51.386280    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:51.389534 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:51.389513    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:51.681081 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:51.680983    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:51.694740 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:51.694713    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:51.711583 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:51.711550    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:52.672423 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:52.672391    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:30:52.675851 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:52.675826    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:30:52.959958 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:52.959928    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:30:52.977651 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:52.977616    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:30:52.994138 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:52.994106    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:30:58.292862 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:58.292830    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-88gv7_467b5b33-2608-4734-8910-d45ba3de5566/global-pull-secret-syncer/0.log"
Jul 27 18:30:58.559884 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:58.559796    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zxf8h_04ea4d62-cc63-43b1-b2a2-3e3cdabbed9c/konnectivity-agent/0.log"
Jul 27 18:30:58.587966 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:30:58.587929    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-194.ec2.internal_1034461a4911a1521e502eca07e06f81/haproxy/0.log"
Jul 27 18:31:03.398290 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:03.398261    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/1.log"
Jul 27 18:31:03.401307 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:03.401282    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-56fb6f6856-n5pb8_73f0dfdd-b806-4dcb-9897-65cf2e003c69/manager/0.log"
Jul 27 18:31:03.751729 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:03.751694    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-fl2ps_fcf0be9b-c592-4fc7-beae-c71f8133f959/manager/0.log"
Jul 27 18:31:03.783169 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:03.783144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-cdzw4_ad29571a-803f-430a-be96-17b11e401dec/limitador/0.log"
Jul 27 18:31:03.824545 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:03.824511    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-774d94476f-6fl6q_f9f5ae01-0ea0-4fa2-98b7-4428d9782d2c/manager/0.log"
Jul 27 18:31:05.757169 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:05.757144    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngjnn_b5d7feb3-e6f9-490e-911e-dc9af38fd78a/node-exporter/0.log"
Jul 27 18:31:05.782550 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:05.782519    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngjnn_b5d7feb3-e6f9-490e-911e-dc9af38fd78a/kube-rbac-proxy/0.log"
Jul 27 18:31:05.810640 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:05.810609    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngjnn_b5d7feb3-e6f9-490e-911e-dc9af38fd78a/init-textfile/0.log"
Jul 27 18:31:07.769520 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:07.769464    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-nw4hs_364f2e29-1e5d-4766-a7bb-4bd6be3a9483/networking-console-plugin/0.log"
Jul 27 18:31:08.314179 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.314141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/2.log"
Jul 27 18:31:08.324087 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.324051    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-d7xqf_5ef78ccd-3ab2-407f-92d3-6ec2ff5e2bcb/console-operator/3.log"
Jul 27 18:31:08.787762 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.787724    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8496f55b4-wjhbs_052e022d-0b92-4f6f-87a7-64dd29328497/console/0.log"
Jul 27 18:31:08.842798 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.842760    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-mbhj4_6dbcbb31-5d54-4be2-ac41-cb4595df7c6e/download-server/0.log"
Jul 27 18:31:08.900833 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.900796    2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"]
Jul 27 18:31:08.901131 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.901119    2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e41e4139-040c-42b7-84cc-e15cf2105d54" containerName="main"
Jul 27 18:31:08.901180 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.901133    2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e41e4139-040c-42b7-84cc-e15cf2105d54" containerName="main"
Jul 27 18:31:08.901219 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.901191    2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e41e4139-040c-42b7-84cc-e15cf2105d54" containerName="main"
Jul 27 18:31:08.903968 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.903950    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:08.906688 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.906652    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5775\"/\"kube-root-ca.crt\""
Jul 27 18:31:08.906688 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.906655    2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q5775\"/\"default-dockercfg-gsljn\""
Jul 27 18:31:08.907665 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.907647    2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q5775\"/\"openshift-service-ca.crt\""
Jul 27 18:31:08.913665 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.913638    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"]
Jul 27 18:31:08.960913 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.960867    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-lib-modules\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:08.960913 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.960911    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-proc\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:08.961148 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.960931    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-podres\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:08.961148 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.961048    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs8hb\" (UniqueName: \"kubernetes.io/projected/9679a1a8-f067-443a-ad12-933cc0d352b6-kube-api-access-cs8hb\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:08.961148 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:08.961121    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-sys\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062295 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062189    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-lib-modules\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062295 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062244    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-proc\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062295 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062272    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-podres\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062606 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs8hb\" (UniqueName: \"kubernetes.io/projected/9679a1a8-f067-443a-ad12-933cc0d352b6-kube-api-access-cs8hb\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062606 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062396    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-proc\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062606 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062407    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-lib-modules\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062606 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062403    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-sys\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062606 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062451    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-podres\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.062606 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.062455    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9679a1a8-f067-443a-ad12-933cc0d352b6-sys\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.070412 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.070388    2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs8hb\" (UniqueName: \"kubernetes.io/projected/9679a1a8-f067-443a-ad12-933cc0d352b6-kube-api-access-cs8hb\") pod \"perf-node-gather-daemonset-6v6cm\" (UID: \"9679a1a8-f067-443a-ad12-933cc0d352b6\") " pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.216727 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.216695    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:09.352416 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.352378    2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"]
Jul 27 18:31:09.355330 ip-10-0-132-194 kubenswrapper[2579]: W0727 18:31:09.355296    2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9679a1a8_f067_443a_ad12_933cc0d352b6.slice/crio-a29af9310725b1039748ce2e7059bb77d712497f4e6dcd30a9880d7e1abd8156 WatchSource:0}: Error finding container a29af9310725b1039748ce2e7059bb77d712497f4e6dcd30a9880d7e1abd8156: Status 404 returned error can't find the container with id a29af9310725b1039748ce2e7059bb77d712497f4e6dcd30a9880d7e1abd8156
Jul 27 18:31:09.470535 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:09.470451    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-q8p9z_c5b6f892-c394-4968-8128-c61aa6f328a2/volume-data-source-validator/0.log"
Jul 27 18:31:10.288123 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.288086    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm" event={"ID":"9679a1a8-f067-443a-ad12-933cc0d352b6","Type":"ContainerStarted","Data":"af250afd316bc87745b73c58fb1baf2aaef5a98673a550e8bec50fdde9d99e73"}
Jul 27 18:31:10.288634 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.288145    2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm" event={"ID":"9679a1a8-f067-443a-ad12-933cc0d352b6","Type":"ContainerStarted","Data":"a29af9310725b1039748ce2e7059bb77d712497f4e6dcd30a9880d7e1abd8156"}
Jul 27 18:31:10.288634 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.288212    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:10.303870 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.303819    2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm" podStartSLOduration=2.303802883 podStartE2EDuration="2.303802883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 18:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 18:31:10.303654338 +0000 UTC m=+5378.889015896" watchObservedRunningTime="2026-07-27 18:31:10.303802883 +0000 UTC m=+5378.889164440"
Jul 27 18:31:10.595150 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.595063    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tzvrv_d51131f9-ae3d-48ce-94b7-ce04477ef8f1/dns/0.log"
Jul 27 18:31:10.621253 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.621228    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tzvrv_d51131f9-ae3d-48ce-94b7-ce04477ef8f1/kube-rbac-proxy/0.log"
Jul 27 18:31:10.647806 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:10.647781    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5brb2_286ed58f-538e-4c7e-8799-310708c75eed/dns-node-resolver/0.log"
Jul 27 18:31:11.283363 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:11.283333    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5hgbt_05546d5f-30c5-44f2-bf23-7bb187cc79b5/node-ca/0.log"
Jul 27 18:31:12.663752 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:12.663714    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:31:12.762793 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:12.762765    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:31:13.310229 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:13.310203    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-c5glj_da55db76-5640-4deb-916f-58b5c50c9f95/serve-healthcheck-canary/0.log"
Jul 27 18:31:13.964337 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:13.964303    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mz5cv_a8fa5de4-86ff-43ff-923e-5a3bf987c540/kube-rbac-proxy/0.log"
Jul 27 18:31:13.992656 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:13.992626    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mz5cv_a8fa5de4-86ff-43ff-923e-5a3bf987c540/exporter/0.log"
Jul 27 18:31:14.017225 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:14.017204    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mz5cv_a8fa5de4-86ff-43ff-923e-5a3bf987c540/extractor/0.log"
Jul 27 18:31:16.302621 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:16.302593    2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q5775/perf-node-gather-daemonset-6v6cm"
Jul 27 18:31:18.905181 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:18.905141    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-bzpsq_773b4faa-3217-4539-8a27-5246848c6be4/seaweedfs/0.log"
Jul 27 18:31:18.981153 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:18.981118    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:31:19.070367 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:19.070335    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:31:20.467173 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:20.467138    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-hkbqw_bd6688e3-515f-47ac-b6c2-d72ebedb5102/discovery/0.log"
Jul 27 18:31:20.556333 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:20.556298    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-57898f8749-pb7sc_dc3e5f58-84bb-4406-b871-1dbdea1e2104/router/0.log"
Jul 27 18:31:25.270431 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.270399    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-5k9gw_122ef338-7a5d-409d-813d-b39955c3044a/kube-multus/0.log"
Jul 27 18:31:25.536355 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.536275    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/kube-multus-additional-cni-plugins/0.log"
Jul 27 18:31:25.557511 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.557451    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/egress-router-binary-copy/0.log"
Jul 27 18:31:25.579237 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.579216    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/cni-plugins/0.log"
Jul 27 18:31:25.601269 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.601240    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/bond-cni-plugin/0.log"
Jul 27 18:31:25.622335 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.622313    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/routeoverride-cni/0.log"
Jul 27 18:31:25.648156 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.648131    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/whereabouts-cni-bincopy/0.log"
Jul 27 18:31:25.670076 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.670053    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-52wdr_652d605f-aa7e-4e83-9227-38245f2c2998/whereabouts-cni/0.log"
Jul 27 18:31:25.957806 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.957770    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v6bp7_158b0a3d-bf31-4102-9fd5-be98f076085c/network-metrics-daemon/0.log"
Jul 27 18:31:25.977502 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:25.977441    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-v6bp7_158b0a3d-bf31-4102-9fd5-be98f076085c/kube-rbac-proxy/0.log"
Jul 27 18:31:26.863736 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:26.863686    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd48h_cb742a4a-419b-4f76-a3b9-06f56c3b4fc6/ovn-controller/0.log"
Jul 27 18:31:26.882841 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:26.882810    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd48h_cb742a4a-419b-4f76-a3b9-06f56c3b4fc6/ovn-acl-logging/0.log"
Jul 27 18:31:26.914629 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:26.914601    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd48h_cb742a4a-419b-4f76-a3b9-06f56c3b4fc6/kube-rbac-proxy-node/0.log"
Jul 27 18:31:26.940331 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:26.940300    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd48h_cb742a4a-419b-4f76-a3b9-06f56c3b4fc6/kube-rbac-proxy-ovn-metrics/0.log"
Jul 27 18:31:26.963122 ip-10-0-132-194 kubenswrapper[2579]: I0727 18:31:26.963097    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bd48h_cb742a4a-419b-4f76-a3b9-06f56c3b4fc6/northd/0.log"