Sep 09 19:54:55.610849 ip-10-0-135-242 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 19:54:56.018466 ip-10-0-135-242 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:56.018466 ip-10-0-135-242 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 19:54:56.018466 ip-10-0-135-242 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:56.018466 ip-10-0-135-242 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 19:54:56.018466 ip-10-0-135-242 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 19:54:56.019929 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.019844    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 19:54:56.023076 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023060    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:56.023076 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023076    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023079    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023083    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023086    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023090    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023093    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023095    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023098    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023101    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023105    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023108    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023111    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023113    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023116    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023119    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023122    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023124    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023127    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023130    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023132    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:56.023142 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023135    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023137    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023140    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023142    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023145    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023148    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023150    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023153    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023156    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023159    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023163    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023167    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023170    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023174    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023177    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023179    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023182    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023185    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023188    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:56.023631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023190    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023193    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023195    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023198    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023201    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023203    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023206    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023208    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023211    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023219    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023222    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023224    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023227    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023229    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023231    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023235    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023238    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023240    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023243    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023246    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:56.024189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023248    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023251    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023253    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023256    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023260    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023263    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023266    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023269    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023271    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023274    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023276    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023279    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023281    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023284    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023286    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023288    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023292    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023295    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023299    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:56.024694 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023303    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023305    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023308    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023310    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023313    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023315    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023318    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023321    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023325    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023328    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023330    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023333    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023335    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023338    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023340    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023343    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023345    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023348    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023350    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:56.025343 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.023353    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025651    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025662    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025666    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025669    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025672    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025675    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025678    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025681    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025684    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025687    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025690    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025693    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025695    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025698    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025701    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025704    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025708    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025713    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025716    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:56.026151 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025719    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025722    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025724    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025727    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025730    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025732    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025735    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025743    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025745    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025748    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025751    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025754    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025757    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025759    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025765    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025769    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025772    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025775    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025778    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:56.026807 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025780    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025783    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025786    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025789    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025791    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025795    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025799    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025802    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025805    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025808    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025811    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025814    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025818    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025820    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025823    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025826    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025829    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025831    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025834    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025836    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:56.027282 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025839    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025841    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025844    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025848    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025851    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025853    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025856    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025860    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025863    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025866    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025869    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025871    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025879    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025881    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025884    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025886    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025889    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025891    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025894    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:56.027782 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025896    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025899    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025901    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025904    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025906    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025909    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025912    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025914    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025917    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025919    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025922    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025924    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025927    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025930    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025932    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025935    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025938    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025941    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025943    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:56.028245 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025946    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025948    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.025951    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028701    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028708    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028711    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028715    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028719    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028721    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028724    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028727    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028731    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028733    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028737    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:56.028734 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028740    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028743    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028747    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028750    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028752    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028757    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028761    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028764    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028766    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028769    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028773    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028777    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028780    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028783    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028786    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028789    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028792    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028794    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028798    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028801    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:56.029078 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028805    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028807    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028812    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028817    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028820    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028823    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028826    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028830    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028833    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028836    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028838    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028840    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028844    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028847    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028851    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028854    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028858    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028862    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028865    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028868    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:56.029565 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028872    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028875    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028878    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028880    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028883    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028887    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028889    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028892    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028894    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028897    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028901    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028904    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028906    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028909    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028913    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028916    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028919    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028922    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028925    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028928    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:56.030057 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028930    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028935    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028938    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028941    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028943    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028946    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028949    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028952    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028955    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028958    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028964    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028968    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028970    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028975    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028977    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028980    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028982    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028985    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028989    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:56.030530 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028991    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028994    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.028997    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029002    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029005    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029007    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029010    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029012    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029015    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029096    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029142    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029149    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029155    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029159    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029163    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029169    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029173    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029177    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029181    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029186    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029191    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 19:54:56.030999 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029194    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029198    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029200    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029203    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029208    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029210    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029215    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029218    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029221    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029225    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029229    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029233    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029236    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029240    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029244    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029248    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029251    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029255    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029259    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029262    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029266    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029270    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029273    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029276    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029279    2576 flags.go:64] FLAG: --enable-server="true"
Sep 09 19:54:56.031505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029283    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029288    2576 flags.go:64] FLAG: --event-burst="100"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029290    2576 flags.go:64] FLAG: --event-qps="50"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029293    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029297    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029302    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029306    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029309    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029313    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029317    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029319    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029322    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029327    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029330    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029333    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029336    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029339    2576 flags.go:64] FLAG: --feature-gates=""
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029344    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029348    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029351    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029355    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029358    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029361    2576 flags.go:64] FLAG: --help="false"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029364    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029368    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 19:54:56.032157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029372    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029375    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029378    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029382    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029386    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029388    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029391    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029394    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029397    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029402    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029405    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029407    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029411    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029415    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029419    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029421    2576 flags.go:64] FLAG: --lock-file=""
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029424    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029429    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029432    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029437    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029442    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029445    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029448    2576 flags.go:64] FLAG: --logging-format="text"
Sep 09 19:54:56.032782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029451    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029456    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029459    2576 flags.go:64] FLAG: --manifest-url=""
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029462    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029471    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029474    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029478    2576 flags.go:64] FLAG: --max-pods="110"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029481    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029484    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029488    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029492    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029494    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029497    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029501    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029509    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029512    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029516    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029521    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029524    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029530    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029534    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029537    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029540    2576 flags.go:64] FLAG: --port="10250"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029543    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 19:54:56.033362 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029547    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a625a669459a5463"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029551    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029553    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029556    2576 flags.go:64] FLAG: --register-node="true"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029561    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029564    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029568    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029571    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029591    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029597    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029600    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029605    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029608    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029611    2576 flags.go:64] FLAG: --runonce="false"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029615    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029619    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029622    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029625    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029629    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029632    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029635    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029639    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029642    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029645    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029648    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029652    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 19:54:56.033962 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029655    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029658    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029662    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029669    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029671    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029675    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029681    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029684    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029688    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029692    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029694    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029699    2576 flags.go:64] FLAG: --v="2"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029703    2576 flags.go:64] FLAG: --version="false"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029708    2576 flags.go:64] FLAG: --vmodule=""
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029712    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.029716    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029843    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029847    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029851    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029854    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029857    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029861    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029864    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:56.034610 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029866    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029869    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029873    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029875    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029879    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029884    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029888    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029892    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029894    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029900    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029904    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029906    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029911    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029915    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029918    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029921    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029924    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029928    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029931    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:56.035159 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029935    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029938    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029941    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029943    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029947    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029950    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029952    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029955    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029958    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029960    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029964    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029967    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029969    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029973    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029977    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029979    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029982    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029986    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029989    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029992    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:56.035674 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029994    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.029998    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030003    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030007    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030011    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030017    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030022    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030025    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030027    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030030    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030033    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030038    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030041    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030046    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030049    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030052    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030056    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030060    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030063    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:56.036169 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030066    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030070    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030073    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030076    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030079    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030082    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030085    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030088    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030091    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030095    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030097    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030100    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030104    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030107    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030110    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030114    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030117    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030121    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030126    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030128    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:56.036670 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030132    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030136    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030139    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030141    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030145    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030148    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030150    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030155    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030160    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030163    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030165    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030169    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030172    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.030174    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:56.037157 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.030179    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:56.037764 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.037743    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 19:54:56.037764 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.037764    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 19:54:56.038586 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.038564    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 19:54:56.038627 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.038590    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 19:54:56.038657 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038650    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:56.038657 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038656    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038661    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038666    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038669    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038672    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038675    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038679    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038682    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038685    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038687    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038690    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038693    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038696    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038699    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038702    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038705    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038708    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038710    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038713    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:56.038715 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038716    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038718    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038722    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038724    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038728    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038731    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038733    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038736    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038739    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038741    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038744    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038747    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038749    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038752    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038754    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038757    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038759    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038762    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038764    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038767    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:56.039189 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038769    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038772    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038775    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038778    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038781    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038783    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038786    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038788    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038791    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038793    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038796    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038798    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038801    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038803    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038806    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038809    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038812    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038815    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038818    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038821    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:56.039693 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038823    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038826    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038828    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038831    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038834    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038836    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038839    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038842    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038844    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038847    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038849    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038852    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038854    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038857    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038860    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038863    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038866    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038869    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038873    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038876    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:56.040201 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038880    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038882    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038885    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038887    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038890    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038893    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038895    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038898    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038901    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038903    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038907    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038909    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038912    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038915    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038917    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038920    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038924    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038928    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:56.040706 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.038930    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.038936    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039048    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039052    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039055    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039058    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039061    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039064    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039068    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039071    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039075    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039078    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039081    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039083    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039086    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 19:54:56.041138 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039089    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039091    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039094    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039097    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039099    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039102    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039105    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039108    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039110    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039113    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039116    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039118    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039121    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039125    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039128    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039131    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039134    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039138    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039140    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 19:54:56.041508 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039143    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039146    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039149    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039152    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039154    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039157    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039160    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039163    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039166    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039169    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039171    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039174    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039176    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039179    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039181    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039184    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039186    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039189    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039192    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039195    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 19:54:56.041990 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039197    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039200    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039203    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039205    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039213    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039216    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039219    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039222    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039224    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039227    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039229    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039232    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039234    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039237    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039240    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039242    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039245    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039248    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039250    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 19:54:56.042467 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039253    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039256    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039258    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039261    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039263    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039266    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039288    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039292    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039295    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039298    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039300    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039303    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039306    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039309    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039313    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039315    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039318    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039320    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039323    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039327    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 19:54:56.042943 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039329    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039332    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039334    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039337    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039340    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039342    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039344    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:56.039347    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.039351    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.039469    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 19:54:56.043417 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.041763    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 19:54:56.043753 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.043742    2576 server.go:1019] "Starting client certificate rotation"
Sep 09 19:54:56.043855 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.043838    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:54:56.043883 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.043878    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 19:54:56.046506 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.046493    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:54:56.051915 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.051898    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 19:54:56.064781 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.064759    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 09 19:54:56.068205 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.068190    2576 log.go:25] "Validated CRI v1 image API"
Sep 09 19:54:56.069049 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.069034    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 19:54:56.071963 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.071944    2576 fs.go:135] Filesystem UUIDs: map[11d80341-6f84-487e-9a8c-24db4131dbe1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f4be4fcf-e9f7-4f45-9643-72f4a3718c7a:/dev/nvme0n1p4]
Sep 09 19:54:56.072021 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.071962    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 19:54:56.075798 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.075780    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:54:56.078333 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.078225    2576 manager.go:216] Machine: {Timestamp:2026-09-09 19:54:56.076561297 +0000 UTC m=+0.339075651 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101650 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ab42337fadbe47cb2ebf90a93442f SystemUUID:ec2ab423-37fa-dbe4-7cb2-ebf90a93442f BootID:49bbdf77-6793-4de2-aebc-dc24747ddf50 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b2:54:e0:dd:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b2:54:e0:dd:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:3d:e9:c6:bf:7e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 19:54:56.078333 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.078333    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 19:54:56.078425 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.078413    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 19:54:56.082134 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.082111    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 19:54:56.082269 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.082137    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-242.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 19:54:56.082959 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.082949    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 19:54:56.082988 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.082961    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 19:54:56.082988 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.082969    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:54:56.083891 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.083881    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 19:54:56.085444 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.085434    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 19:54:56.090517 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090504    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:54:56.090665 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090655    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 19:54:56.090697 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090687    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 19:54:56.090697 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090696    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 19:54:56.090751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090710    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 19:54:56.090751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090719    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 09 19:54:56.090751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.090727    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 19:54:56.091079 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.091068    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:54:56.091115 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.091087    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 19:54:56.093493 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.093473    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 19:54:56.096505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096490    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 19:54:56.096551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096520    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 19:54:56.096681 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096670    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 19:54:56.096741 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096685    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 19:54:56.096741 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096690    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 19:54:56.096741 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096696    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 19:54:56.096741 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096703    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 19:54:56.096741 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.096708    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 19:54:56.097379 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.097370    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 19:54:56.097379 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.097378    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 19:54:56.097478 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.097385    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 19:54:56.097478 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.097391    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 19:54:56.097478 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.097399    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 19:54:56.097478 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.097407    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 19:54:56.098216 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.098207    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 19:54:56.098216 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.098215    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 19:54:56.100868 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.100852    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kn22c"
Sep 09 19:54:56.101916 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.101904    2576 server.go:1268] "Started kubelet"
Sep 09 19:54:56.102024 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.101997    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 19:54:56.102074 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.101990    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 19:54:56.102074 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.102066    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 19:54:56.102752 ip-10-0-135-242 systemd[1]: Started Kubernetes Kubelet.
Sep 09 19:54:56.107343 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.103602    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 19:54:56.109050 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.109027    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kn22c"
Sep 09 19:54:56.109422 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.109402    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 19:54:56.110397 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.110374    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 19:54:56.110639 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.110612    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-242.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 19:54:56.110809 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.110781    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 19:54:56.110923 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.110907    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 19:54:56.113309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.113281    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 19:54:56.113609 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.110658    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-242.ec2.internal.18d3bf31731e2199  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-242.ec2.internal,UID:ip-10-0-135-242.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-242.ec2.internal,},FirstTimestamp:2026-09-09 19:54:56.101876121 +0000 UTC m=+0.364390476,LastTimestamp:2026-09-09 19:54:56.101876121 +0000 UTC m=+0.364390476,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-242.ec2.internal,}"
Sep 09 19:54:56.114330 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.114315    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 19:54:56.114656 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.114634    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.115197 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115179    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 19:54:56.115197 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115185    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 19:54:56.115339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115207    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 19:54:56.115339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115309    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 19:54:56.115339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115319    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 19:54:56.115663 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115643    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 19:54:56.115663 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115664    2576 factory.go:55] Registering systemd factory
Sep 09 19:54:56.115752 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115674    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 09 19:54:56.115900 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115889    2576 factory.go:153] Registering CRI-O factory
Sep 09 19:54:56.115941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115902    2576 factory.go:223] Registration of the crio container factory successfully
Sep 09 19:54:56.115941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115915    2576 factory.go:103] Registering Raw factory
Sep 09 19:54:56.115941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.115928    2576 manager.go:1196] Started watching for new ooms in manager
Sep 09 19:54:56.116691 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.116665    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-242.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 19:54:56.116691 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.116678    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 19:54:56.116892 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.116879    2576 manager.go:349] Starting recovery of all containers
Sep 09 19:54:56.127267 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.127252    2576 manager.go:354] Recovery completed
Sep 09 19:54:56.131904 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.131891    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.134647 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.134631    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.134708 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.134666    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.134708 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.134682    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.135293 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.135278    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 19:54:56.135355 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.135293    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.135355 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.135292    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 19:54:56.135426 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.135377    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 19:54:56.136916 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.136901    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.137004 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.136930    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.137004 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.136944    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.137620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.137607    2576 policy_none.go:49] "None policy: Start"
Sep 09 19:54:56.137689 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.137625    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 19:54:56.137689 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.137638    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 19:54:56.139666 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.139652    2576 policy_none.go:47] "Start"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178106    2576 manager.go:337] "Starting Device Plugin manager"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.178145    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178155    2576 server.go:87] "Starting device plugin registration server"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178414    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178425    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178655    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178731    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.178740    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.179273    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 19:54:56.194309 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.179342    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.210000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.209969    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 19:54:56.211284 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.211267    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 19:54:56.211385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.211313    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 19:54:56.211385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.211336    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 19:54:56.211385 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.211376    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 09 19:54:56.213873 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.213853    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:56.279232 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.279125    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.281811 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.280932    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.281811 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.281005    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.281811 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.281022    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.281811 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.281058    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.288697 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.288680    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.288754 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.288704    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-242.ec2.internal\": node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.303179 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.303156    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.312730 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.312194    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"]
Sep 09 19:54:56.312813 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.312768    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.313625 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.313609    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.313692 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.313638    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.313692 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.313651    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.315842 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.315829    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.315963 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.315950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.315998 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.315977    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.316045 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316031    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.316079 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.316615 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316600    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.316680 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316633    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.316680 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316641    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.316680 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316648    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.316680 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316660    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.316680 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.316671    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.318681 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.318664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.318757 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.318689    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 19:54:56.319849 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.319835    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 19:54:56.319932 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.319855    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 19:54:56.319932 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.319867    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeHasSufficientPID"
Sep 09 19:54:56.336115 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.336094    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.339521 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.339502    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-242.ec2.internal\" not found" node="ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.403721 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.403693    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.416629 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.416606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c620202158ca74d37111884761e47e70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"c620202158ca74d37111884761e47e70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.416704 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.416641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.416704 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.416658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.416770 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.416721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.416770 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.416721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c1647d1d87b8b8396e8fceb50173ea2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal\" (UID: \"8c1647d1d87b8b8396e8fceb50173ea2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.504234 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.504206    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.517672 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.517641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c620202158ca74d37111884761e47e70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"c620202158ca74d37111884761e47e70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.517732 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.517700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c620202158ca74d37111884761e47e70-config\") pod \"kube-apiserver-proxy-ip-10-0-135-242.ec2.internal\" (UID: \"c620202158ca74d37111884761e47e70\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.604440 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.604416    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.638934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.638900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.642008 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:56.641987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:56.705274 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.705243    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.805746 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.805715    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:56.906323 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:56.906254    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:57.006782 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:57.006745    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:57.043264 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:57.043238    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 19:54:57.043685 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:57.043375    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 19:54:57.107723 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:57.107695    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:57.115638 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:57.115612    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 19:49:57 +0000 UTC" deadline="2028-05-24 18:35:45.4876868 +0000 UTC"
Sep 09 19:54:57.115683 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:57.115636    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14950h40m48.372053855s"
Sep 09 19:54:57.208826 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:57.208755    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:57.256782 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:57.256757    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:57.308972 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:57.308943    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:59.153927 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.153902    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:59.210914 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.210848    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:59.254715 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.254687    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-242.ec2.internal\" not found"
Sep 09 19:54:59.287831 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:59.287788    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc620202158ca74d37111884761e47e70.slice/crio-b3ac3fd2bc3be4c69e2cd28a7546a4e4dc2b20e5da990066af0773cda90ed2f4 WatchSource:0}: Error finding container b3ac3fd2bc3be4c69e2cd28a7546a4e4dc2b20e5da990066af0773cda90ed2f4: Status 404 returned error can't find the container with id b3ac3fd2bc3be4c69e2cd28a7546a4e4dc2b20e5da990066af0773cda90ed2f4
Sep 09 19:54:59.288097 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:54:59.288079    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1647d1d87b8b8396e8fceb50173ea2.slice/crio-2239758e98a07a958324212d55cf4252c2619910c9ce5be513d5c37f298269a2 WatchSource:0}: Error finding container 2239758e98a07a958324212d55cf4252c2619910c9ce5be513d5c37f298269a2: Status 404 returned error can't find the container with id 2239758e98a07a958324212d55cf4252c2619910c9ce5be513d5c37f298269a2
Sep 09 19:54:59.290941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.290924    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:59.293169 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.293145    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 19:54:59.359786 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.359622    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:59.371850 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.371831    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:54:59.372935 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.372923    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal"
Sep 09 19:54:59.385584 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.385567    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 19:54:59.836117 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.836078    2576 apiserver.go:52] "Watching apiserver"
Sep 09 19:54:59.842493 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.842469    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 19:54:59.844736 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.844705    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal","openshift-multus/network-metrics-daemon-spnhz","openshift-network-diagnostics/network-check-target-gpmgb","openshift-network-operator/iptables-alerter-dddng","openshift-ovn-kubernetes/ovnkube-node-dz5df","kube-system/global-pull-secret-syncer-mc9xw","kube-system/konnectivity-agent-t9j5m","kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal","openshift-cluster-node-tuning-operator/tuned-92f2s","openshift-dns/node-resolver-c7sbj","openshift-multus/multus-additional-cni-plugins-sc8xs","openshift-multus/multus-vbmbt","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf","openshift-image-registry/node-ca-6xnjs"]
Sep 09 19:54:59.847240 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.847222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.849734 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.849717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-j6zfh\""
Sep 09 19:54:59.849896 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.849884    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 19:54:59.850091 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.850069    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.850519 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.850449    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 19:54:59.852596 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.852578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:54:59.852847 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.852806    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:54:59.853153 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.853134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 19:54:59.853718 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.853508    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.853718 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.853645    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pmv9x\""
Sep 09 19:54:59.853718 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.853672    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.854676 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.854653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.856991 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.856946    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.857092 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.857012    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rqbsj\""
Sep 09 19:54:59.857092 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.857019    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.857898 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.857232    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 19:54:59.857898 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.857316    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.857898 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.857875    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 19:54:59.859746 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.859729    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bc7pc\""
Sep 09 19:54:59.860064 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.860049    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 19:54:59.860228 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.860216    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 19:54:59.860401 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.860386    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.860575 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.860562    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.860833 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.860820    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 19:54:59.860992 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.860974    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 19:54:59.862409 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.862392    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.862506 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.862453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:54:59.862642 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.862626    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.864356 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.864338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:54:59.864855 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.864824    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.864945 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.864902    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fqtvb\""
Sep 09 19:54:59.865091 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.865076    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.866504 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.866486    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.866572 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.866511    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.866572 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.866537    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 19:54:59.867047 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.867032    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lrdct\""
Sep 09 19:54:59.867164 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.867124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:54:59.867223 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.867193    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:54:59.869346 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.869327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:54:59.870142 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-systemd-units\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870240 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870151    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-ovn\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870240 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-systemd\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870240 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870180    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 19:54:59.870240 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-run\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9dsz\" (UniqueName: \"kubernetes.io/projected/2a7c2517-8326-4338-ab93-657f631befac-kube-api-access-m9dsz\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-modprobe-d\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870287    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysconfig\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-var-lib-kubelet\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a7c2517-8326-4338-ab93-657f631befac-etc-tuned\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-device-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.870436 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870413    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f736b1b-7aa0-4509-a18b-c25c0ff131b6-konnectivity-ca\") pod \"konnectivity-agent-t9j5m\" (UID: \"1f736b1b-7aa0-4509-a18b-c25c0ff131b6\") " pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-run-netns\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-ovnkube-config\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f88jn\" (UniqueName: \"kubernetes.io/projected/b6965512-4260-4621-9640-2e7b38066a97-kube-api-access-f88jn\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-kubernetes\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-dbus\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870603    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-node-log\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-run-ovn-kubernetes\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-sys\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmr82\" (UniqueName: \"kubernetes.io/projected/044e22d3-2279-402a-bb91-e543cd59575f-kube-api-access-wmr82\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.870723 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-kubelet-config\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-env-overrides\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6965512-4260-4621-9640-2e7b38066a97-ovn-node-metrics-cert\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-slash\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-etc-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-lib-modules\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.870995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ebec34a-6899-4e62-865a-950c94c88c99-host-slash\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871017    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw9mm\" (UniqueName: \"kubernetes.io/projected/8ebec34a-6899-4e62-865a-950c94c88c99-kube-api-access-xw9mm\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-cni-bin\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-cni-netd\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-host\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:54:59.871167 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-var-lib-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-log-socket\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysctl-d\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f736b1b-7aa0-4509-a18b-c25c0ff131b6-agent-certs\") pod \"konnectivity-agent-t9j5m\" (UID: \"1f736b1b-7aa0-4509-a18b-c25c0ff131b6\") " pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-kubelet\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-ovnkube-script-lib\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysctl-conf\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7c2517-8326-4338-ab93-657f631befac-tmp\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-socket-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-registration-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-sys-fs\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871470    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-systemd\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ebec34a-6899-4e62-865a-950c94c88c99-iptables-alerter-script\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871561    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jzvvv\""
Sep 09 19:54:59.871744 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871736    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.872214 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.871797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.872214 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.872064    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.874434 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.874434 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874430    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 19:54:59.874648 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874632    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 19:54:59.874830 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874814    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lsb5l\""
Sep 09 19:54:59.874892 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874847    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 19:54:59.874892 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874856    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 19:54:59.875030 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.874948    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 19:54:59.876666 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.876647    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 19:54:59.876749 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.876726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rfvlk\""
Sep 09 19:54:59.889827 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.889804    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n5cc5"
Sep 09 19:54:59.899373 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.899355    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n5cc5"
Sep 09 19:54:59.960348 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.960325    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 19:54:59.960829 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.960791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"8c1647d1d87b8b8396e8fceb50173ea2","Type":"ContainerStarted","Data":"2239758e98a07a958324212d55cf4252c2619910c9ce5be513d5c37f298269a2"}
Sep 09 19:54:59.961930 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.961905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" event={"ID":"c620202158ca74d37111884761e47e70","Type":"ContainerStarted","Data":"b3ac3fd2bc3be4c69e2cd28a7546a4e4dc2b20e5da990066af0773cda90ed2f4"}
Sep 09 19:54:59.972445 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f736b1b-7aa0-4509-a18b-c25c0ff131b6-konnectivity-ca\") pod \"konnectivity-agent-t9j5m\" (UID: \"1f736b1b-7aa0-4509-a18b-c25c0ff131b6\") " pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.972542 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f88jn\" (UniqueName: \"kubernetes.io/projected/b6965512-4260-4621-9640-2e7b38066a97-kube-api-access-f88jn\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.972542 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972483    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-dbus\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.972542 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b05865d-a410-494c-803b-b6f4f63195fc-hosts-file\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:54:59.972542 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-daemon-config\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.972722 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-socket-dir-parent\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.972722 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-cni-bin\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.972722 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-env-overrides\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.972722 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972635    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6965512-4260-4621-9640-2e7b38066a97-ovn-node-metrics-cert\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.972722 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.972722 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-dbus\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-etc-kubernetes\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-etc-selinux\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw9mm\" (UniqueName: \"kubernetes.io/projected/8ebec34a-6899-4e62-865a-950c94c88c99-kube-api-access-xw9mm\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9mgr\" (UniqueName: \"kubernetes.io/projected/5731e2df-b062-43b0-9921-5fdf25a9403a-kube-api-access-r9mgr\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.972989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-var-lib-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-log-socket\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-var-lib-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysctl-d\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973134    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-log-socket\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-cni-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysctl-d\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f736b1b-7aa0-4509-a18b-c25c0ff131b6-agent-certs\") pod \"konnectivity-agent-t9j5m\" (UID: \"1f736b1b-7aa0-4509-a18b-c25c0ff131b6\") " pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-kubelet\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysctl-conf\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7c2517-8326-4338-ab93-657f631befac-tmp\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973282    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-kubelet\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-socket-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-env-overrides\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-registration-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzrrj\" (UniqueName: \"kubernetes.io/projected/5b9f9f07-7d47-4c98-a206-f1273c62974e-kube-api-access-bzrrj\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysctl-conf\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973409    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-registration-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-device-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973464    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-device-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-systemd\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-socket-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-systemd-units\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-systemd-units\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973576    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-systemd\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9dsz\" (UniqueName: \"kubernetes.io/projected/2a7c2517-8326-4338-ab93-657f631befac-kube-api-access-m9dsz\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-systemd\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b9f9f07-7d47-4c98-a206-f1273c62974e-host\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:54:59.973941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-cni-multus\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-systemd\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf9p9\" (UniqueName: \"kubernetes.io/projected/08532342-baae-47cc-acf7-ad5929ed0c38-kube-api-access-hf9p9\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5731e2df-b062-43b0-9921-5fdf25a9403a-cni-binary-copy\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-run-netns\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1f736b1b-7aa0-4509-a18b-c25c0ff131b6-konnectivity-ca\") pod \"konnectivity-agent-t9j5m\" (UID: \"1f736b1b-7aa0-4509-a18b-c25c0ff131b6\") " pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973788    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-run-netns\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-ovnkube-config\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-kubernetes\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-netns\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-node-log\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973969    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-node-log\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.973972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-run-ovn-kubernetes\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-sys\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-run-ovn-kubernetes\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-kubernetes\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.974751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmr82\" (UniqueName: \"kubernetes.io/projected/044e22d3-2279-402a-bb91-e543cd59575f-kube-api-access-wmr82\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-sys\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974188    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-kubelet-config\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-kubelet-config\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvgf\" (UniqueName: \"kubernetes.io/projected/c21d1112-0d32-4c3e-9e79-15322990e4a5-kube-api-access-pzvgf\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-kubelet\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-slash\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974342    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-etc-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974379    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-slash\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-lib-modules\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-system-cni-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974410    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-etc-openvswitch\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-k8s-cni-cncf-io\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.975518 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-hostroot\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-lib-modules\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974495    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-ovnkube-config\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ebec34a-6899-4e62-865a-950c94c88c99-host-slash\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-cni-bin\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ebec34a-6899-4e62-865a-950c94c88c99-host-slash\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-cni-netd\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-cni-bin\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-host\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-host-cni-netd\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974704    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-host\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzb7\" (UniqueName: \"kubernetes.io/projected/7b05865d-a410-494c-803b-b6f4f63195fc-kube-api-access-cdzb7\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-multus-certs\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974799    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974808    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-cnibin\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.976314 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974845    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-conf-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.974910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-ovnkube-script-lib\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-sys-fs\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b05865d-a410-494c-803b-b6f4f63195fc-tmp-dir\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b9f9f07-7d47-4c98-a206-f1273c62974e-serviceca\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-cnibin\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ebec34a-6899-4e62-865a-950c94c88c99-iptables-alerter-script\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-ovn\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/044e22d3-2279-402a-bb91-e543cd59575f-sys-fs\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-run\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-modprobe-d\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b6965512-4260-4621-9640-2e7b38066a97-run-ovn\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysconfig\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975299    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-var-lib-kubelet\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975368    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-var-lib-kubelet\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a7c2517-8326-4338-ab93-657f631befac-etc-tuned\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.976862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975402    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-run\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-modprobe-d\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b6965512-4260-4621-9640-2e7b38066a97-ovnkube-script-lib\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975595    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/2a7c2517-8326-4338-ab93-657f631befac-etc-sysconfig\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.975603    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.975769    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:00.475733762 +0000 UTC m=+2.993871289 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.975795    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ebec34a-6899-4e62-865a-950c94c88c99-iptables-alerter-script\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.977454    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7c2517-8326-4338-ab93-657f631befac-tmp\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.977551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.977545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b6965512-4260-4621-9640-2e7b38066a97-ovn-node-metrics-cert\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:54:59.977934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.977915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1f736b1b-7aa0-4509-a18b-c25c0ff131b6-agent-certs\") pod \"konnectivity-agent-t9j5m\" (UID: \"1f736b1b-7aa0-4509-a18b-c25c0ff131b6\") " pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:54:59.980997 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.980981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/2a7c2517-8326-4338-ab93-657f631befac-etc-tuned\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.981982 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.981947    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:54:59.981982 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.981985    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:54:59.981982 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.982000    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:59.982243 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:54:59.982057    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:00.482039722 +0000 UTC m=+3.000177233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:54:59.984577 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.984558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9dsz\" (UniqueName: \"kubernetes.io/projected/2a7c2517-8326-4338-ab93-657f631befac-kube-api-access-m9dsz\") pod \"tuned-92f2s\" (UID: \"2a7c2517-8326-4338-ab93-657f631befac\") " pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:54:59.984842 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.984819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw9mm\" (UniqueName: \"kubernetes.io/projected/8ebec34a-6899-4e62-865a-950c94c88c99-kube-api-access-xw9mm\") pod \"iptables-alerter-dddng\" (UID: \"8ebec34a-6899-4e62-865a-950c94c88c99\") " pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:54:59.987768 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.985178    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmr82\" (UniqueName: \"kubernetes.io/projected/044e22d3-2279-402a-bb91-e543cd59575f-kube-api-access-wmr82\") pod \"aws-ebs-csi-driver-node-vvscf\" (UID: \"044e22d3-2279-402a-bb91-e543cd59575f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:54:59.988417 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.988395    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 19:54:59.988512 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:54:59.988424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f88jn\" (UniqueName: \"kubernetes.io/projected/b6965512-4260-4621-9640-2e7b38066a97-kube-api-access-f88jn\") pod \"ovnkube-node-dz5df\" (UID: \"b6965512-4260-4621-9640-2e7b38066a97\") " pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:00.076382 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.076382 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9mgr\" (UniqueName: \"kubernetes.io/projected/5731e2df-b062-43b0-9921-5fdf25a9403a-kube-api-access-r9mgr\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-cni-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzrrj\" (UniqueName: \"kubernetes.io/projected/5b9f9f07-7d47-4c98-a206-f1273c62974e-kube-api-access-bzrrj\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b9f9f07-7d47-4c98-a206-f1273c62974e-host\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-cni-multus\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf9p9\" (UniqueName: \"kubernetes.io/projected/08532342-baae-47cc-acf7-ad5929ed0c38-kube-api-access-hf9p9\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076637    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-cni-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.076651    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5b9f9f07-7d47-4c98-a206-f1273c62974e-host\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076702    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076710    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-cni-multus\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.076726    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:00.576705776 +0000 UTC m=+3.094843281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5731e2df-b062-43b0-9921-5fdf25a9403a-cni-binary-copy\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076767    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-system-cni-dir\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-netns\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvgf\" (UniqueName: \"kubernetes.io/projected/c21d1112-0d32-4c3e-9e79-15322990e4a5-kube-api-access-pzvgf\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076837    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-netns\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-kubelet\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-system-cni-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-k8s-cni-cncf-io\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-hostroot\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.076981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076976    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-kubelet\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.076988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzb7\" (UniqueName: \"kubernetes.io/projected/7b05865d-a410-494c-803b-b6f4f63195fc-kube-api-access-cdzb7\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077041    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-cni-binary-copy\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-multus-certs\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077070    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-system-cni-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-hostroot\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-k8s-cni-cncf-io\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-run-multus-certs\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077103    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-cnibin\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-conf-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b05865d-a410-494c-803b-b6f4f63195fc-tmp-dir\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b9f9f07-7d47-4c98-a206-f1273c62974e-serviceca\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5731e2df-b062-43b0-9921-5fdf25a9403a-cni-binary-copy\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-cnibin\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077199    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-conf-dir\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077221    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-cnibin\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c21d1112-0d32-4c3e-9e79-15322990e4a5-cnibin\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.077847 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077264    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b05865d-a410-494c-803b-b6f4f63195fc-hosts-file\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-daemon-config\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-socket-dir-parent\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-cni-bin\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7b05865d-a410-494c-803b-b6f4f63195fc-hosts-file\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-etc-kubernetes\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-etc-kubernetes\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077448    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-host-var-lib-cni-bin\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b05865d-a410-494c-803b-b6f4f63195fc-tmp-dir\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-socket-dir-parent\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5b9f9f07-7d47-4c98-a206-f1273c62974e-serviceca\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077724    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5731e2df-b062-43b0-9921-5fdf25a9403a-multus-daemon-config\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.078385 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.077824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c21d1112-0d32-4c3e-9e79-15322990e4a5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.087435 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.087376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9mgr\" (UniqueName: \"kubernetes.io/projected/5731e2df-b062-43b0-9921-5fdf25a9403a-kube-api-access-r9mgr\") pod \"multus-vbmbt\" (UID: \"5731e2df-b062-43b0-9921-5fdf25a9403a\") " pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.088457 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.088434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvgf\" (UniqueName: \"kubernetes.io/projected/c21d1112-0d32-4c3e-9e79-15322990e4a5-kube-api-access-pzvgf\") pod \"multus-additional-cni-plugins-sc8xs\" (UID: \"c21d1112-0d32-4c3e-9e79-15322990e4a5\") " pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.088559 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.088496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzb7\" (UniqueName: \"kubernetes.io/projected/7b05865d-a410-494c-803b-b6f4f63195fc-kube-api-access-cdzb7\") pod \"node-resolver-c7sbj\" (UID: \"7b05865d-a410-494c-803b-b6f4f63195fc\") " pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.088676 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.088656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzrrj\" (UniqueName: \"kubernetes.io/projected/5b9f9f07-7d47-4c98-a206-f1273c62974e-kube-api-access-bzrrj\") pod \"node-ca-6xnjs\" (UID: \"5b9f9f07-7d47-4c98-a206-f1273c62974e\") " pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.088716 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.088680    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf9p9\" (UniqueName: \"kubernetes.io/projected/08532342-baae-47cc-acf7-ad5929ed0c38-kube-api-access-hf9p9\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:00.162150 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.162111    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:55:00.171066 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.171045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dddng"
Sep 09 19:55:00.177882 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.177864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf"
Sep 09 19:55:00.185563 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.185546    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:00.192226 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.192201    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-92f2s"
Sep 09 19:55:00.197902 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.197876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6xnjs"
Sep 09 19:55:00.204442 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.204426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c7sbj"
Sep 09 19:55:00.212095 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.212080    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sc8xs"
Sep 09 19:55:00.214819 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.214789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vbmbt"
Sep 09 19:55:00.439555 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.439521    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b05865d_a410_494c_803b_b6f4f63195fc.slice/crio-548f2b976bcd0bc3df834474d7831f2b6f56aed97b975cf68a7dd56e3ad0f332 WatchSource:0}: Error finding container 548f2b976bcd0bc3df834474d7831f2b6f56aed97b975cf68a7dd56e3ad0f332: Status 404 returned error can't find the container with id 548f2b976bcd0bc3df834474d7831f2b6f56aed97b975cf68a7dd56e3ad0f332
Sep 09 19:55:00.440116 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.440084    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebec34a_6899_4e62_865a_950c94c88c99.slice/crio-1802357134a4d29ae665a4b1431f3cfca9a8bca2c606f272471ce285ac1353cf WatchSource:0}: Error finding container 1802357134a4d29ae665a4b1431f3cfca9a8bca2c606f272471ce285ac1353cf: Status 404 returned error can't find the container with id 1802357134a4d29ae665a4b1431f3cfca9a8bca2c606f272471ce285ac1353cf
Sep 09 19:55:00.443195 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.443165    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc21d1112_0d32_4c3e_9e79_15322990e4a5.slice/crio-a0ee9fe72332387640dab547035c1dee8a9923b0e4c588d3f10d8f816fe873ec WatchSource:0}: Error finding container a0ee9fe72332387640dab547035c1dee8a9923b0e4c588d3f10d8f816fe873ec: Status 404 returned error can't find the container with id a0ee9fe72332387640dab547035c1dee8a9923b0e4c588d3f10d8f816fe873ec
Sep 09 19:55:00.446085 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.446063    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5731e2df_b062_43b0_9921_5fdf25a9403a.slice/crio-9a441f65d9b382bbc386145abf5fdf49144de482b67e4aa0a1502907eec91b8d WatchSource:0}: Error finding container 9a441f65d9b382bbc386145abf5fdf49144de482b67e4aa0a1502907eec91b8d: Status 404 returned error can't find the container with id 9a441f65d9b382bbc386145abf5fdf49144de482b67e4aa0a1502907eec91b8d
Sep 09 19:55:00.449095 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.447075    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f736b1b_7aa0_4509_a18b_c25c0ff131b6.slice/crio-b47ddc4acf9cb2932ec0c6e640dd6f645f3fa3ab76af0596a1949e3ef747e461 WatchSource:0}: Error finding container b47ddc4acf9cb2932ec0c6e640dd6f645f3fa3ab76af0596a1949e3ef747e461: Status 404 returned error can't find the container with id b47ddc4acf9cb2932ec0c6e640dd6f645f3fa3ab76af0596a1949e3ef747e461
Sep 09 19:55:00.468631 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.468592    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6965512_4260_4621_9640_2e7b38066a97.slice/crio-6ec595df62ae57c4aebcc1a0e808c049397dc133fff20bff9e64f4654d8bdd2a WatchSource:0}: Error finding container 6ec595df62ae57c4aebcc1a0e808c049397dc133fff20bff9e64f4654d8bdd2a: Status 404 returned error can't find the container with id 6ec595df62ae57c4aebcc1a0e808c049397dc133fff20bff9e64f4654d8bdd2a
Sep 09 19:55:00.469347 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.469321    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7c2517_8326_4338_ab93_657f631befac.slice/crio-e8905955050389f3f6f892c380baf419ad4e7c799614c1409940961c647938ad WatchSource:0}: Error finding container e8905955050389f3f6f892c380baf419ad4e7c799614c1409940961c647938ad: Status 404 returned error can't find the container with id e8905955050389f3f6f892c380baf419ad4e7c799614c1409940961c647938ad
Sep 09 19:55:00.470386 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.470269    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9f9f07_7d47_4c98_a206_f1273c62974e.slice/crio-db472e98a84448128ca5f07127c9c01bed6f7f999e0ef917d35156262d6ef708 WatchSource:0}: Error finding container db472e98a84448128ca5f07127c9c01bed6f7f999e0ef917d35156262d6ef708: Status 404 returned error can't find the container with id db472e98a84448128ca5f07127c9c01bed6f7f999e0ef917d35156262d6ef708
Sep 09 19:55:00.471285 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:00.471265    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod044e22d3_2279_402a_bb91_e543cd59575f.slice/crio-ee07b7791742c30a666f899418bc3f6578b74fa9714e7eb57afa4bcb327ac50a WatchSource:0}: Error finding container ee07b7791742c30a666f899418bc3f6578b74fa9714e7eb57afa4bcb327ac50a: Status 404 returned error can't find the container with id ee07b7791742c30a666f899418bc3f6578b74fa9714e7eb57afa4bcb327ac50a
Sep 09 19:55:00.480023 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.479970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:00.480093 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.480064    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:00.480139 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.480108    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:01.480095247 +0000 UTC m=+3.998232756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:00.581340 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.581301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:00.581486 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.581372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:00.581486 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.581461    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:00.581486 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.581463    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:00.581486 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.581485    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:00.581634 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.581505    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:00.581634 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.581524    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:01.581505411 +0000 UTC m=+4.099642931 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:00.581634 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.581541    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:01.581530415 +0000 UTC m=+4.099667938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:00.902319 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.902257    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:49:59 +0000 UTC" deadline="2028-04-24 06:59:22.439478083 +0000 UTC"
Sep 09 19:55:00.902319 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.902315    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14219h4m21.537167059s"
Sep 09 19:55:00.956866 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.956832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:00.957296 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:00.957265    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:00.972996 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.972936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf" event={"ID":"044e22d3-2279-402a-bb91-e543cd59575f","Type":"ContainerStarted","Data":"ee07b7791742c30a666f899418bc3f6578b74fa9714e7eb57afa4bcb327ac50a"}
Sep 09 19:55:00.974927 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.974781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-92f2s" event={"ID":"2a7c2517-8326-4338-ab93-657f631befac","Type":"ContainerStarted","Data":"e8905955050389f3f6f892c380baf419ad4e7c799614c1409940961c647938ad"}
Sep 09 19:55:00.976981 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.976938    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbmbt" event={"ID":"5731e2df-b062-43b0-9921-5fdf25a9403a","Type":"ContainerStarted","Data":"9a441f65d9b382bbc386145abf5fdf49144de482b67e4aa0a1502907eec91b8d"}
Sep 09 19:55:00.979160 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.979129    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerStarted","Data":"a0ee9fe72332387640dab547035c1dee8a9923b0e4c588d3f10d8f816fe873ec"}
Sep 09 19:55:00.982121 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.982092    2576 generic.go:356] "Generic (PLEG): container finished" podID="8c1647d1d87b8b8396e8fceb50173ea2" containerID="3a0d326c17a1a2031e8113e57503648968e8bd167c660fe094f3eef15abad22b" exitCode=0
Sep 09 19:55:00.982232 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.982175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"8c1647d1d87b8b8396e8fceb50173ea2","Type":"ContainerDied","Data":"3a0d326c17a1a2031e8113e57503648968e8bd167c660fe094f3eef15abad22b"}
Sep 09 19:55:00.984825 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.984763    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6xnjs" event={"ID":"5b9f9f07-7d47-4c98-a206-f1273c62974e","Type":"ContainerStarted","Data":"db472e98a84448128ca5f07127c9c01bed6f7f999e0ef917d35156262d6ef708"}
Sep 09 19:55:00.987114 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.987010    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"6ec595df62ae57c4aebcc1a0e808c049397dc133fff20bff9e64f4654d8bdd2a"}
Sep 09 19:55:00.990556 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.990494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t9j5m" event={"ID":"1f736b1b-7aa0-4509-a18b-c25c0ff131b6","Type":"ContainerStarted","Data":"b47ddc4acf9cb2932ec0c6e640dd6f645f3fa3ab76af0596a1949e3ef747e461"}
Sep 09 19:55:00.993910 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:00.993845    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dddng" event={"ID":"8ebec34a-6899-4e62-865a-950c94c88c99","Type":"ContainerStarted","Data":"1802357134a4d29ae665a4b1431f3cfca9a8bca2c606f272471ce285ac1353cf"}
Sep 09 19:55:01.004661 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.004627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c7sbj" event={"ID":"7b05865d-a410-494c-803b-b6f4f63195fc","Type":"ContainerStarted","Data":"548f2b976bcd0bc3df834474d7831f2b6f56aed97b975cf68a7dd56e3ad0f332"}
Sep 09 19:55:01.489238 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.488675    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:01.489238 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.488837    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:01.489238 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.488902    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:03.488884333 +0000 UTC m=+6.007021839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:01.589998 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.589894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:01.589998 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.589984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:01.590297 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.590154    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:01.590297 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.590173    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:01.590297 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.590186    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:01.590297 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.590246    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:03.590228222 +0000 UTC m=+6.108365731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:01.590671 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.590650    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:01.590749 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.590731    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:03.590692166 +0000 UTC m=+6.108829675 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:01.903313 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.903271    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 19:49:59 +0000 UTC" deadline="2028-03-12 13:54:12.482045486 +0000 UTC"
Sep 09 19:55:01.903313 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.903311    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13193h59m10.578738908s"
Sep 09 19:55:01.959594 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.959065    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:01.959594 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.959189    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:01.960146 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:01.959906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:01.960146 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:01.960031    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:02.028076 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:02.028030    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" event={"ID":"8c1647d1d87b8b8396e8fceb50173ea2","Type":"ContainerStarted","Data":"603dc405cdba05f66fd22b6cec3053d29355234169911458dbb1c7f7883d12af"}
Sep 09 19:55:02.956860 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:02.956324    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:02.956860 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:02.956484    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:03.505159 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:03.505085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:03.505353 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.505241    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:03.505353 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.505316    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:07.505296573 +0000 UTC m=+10.023434084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:03.607617 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:03.607571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:03.607797 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:03.607660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:03.607866 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.607815    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:03.607866 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.607835    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:03.607866 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.607847    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:03.608042 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.607900    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:07.607885231 +0000 UTC m=+10.126022739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:03.608139 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.608094    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:03.608198 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.608168    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:07.608148704 +0000 UTC m=+10.126286223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:03.958670 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:03.956582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:03.958670 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.956723    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:03.958670 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:03.957164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:03.958670 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:03.957256    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:04.957441 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:04.956885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:04.957441 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:04.957048    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:05.957082 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:05.957050    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:05.957602 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:05.957189    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:05.957602 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:05.957284    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:05.957602 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:05.957410    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:06.956716 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:06.956678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:06.956893 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:06.956836    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:07.538384 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:07.538340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:07.538850 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.538522    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:07.538850 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.538596    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:15.538576127 +0000 UTC m=+18.056713635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:07.638927 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:07.638884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:07.639108 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:07.638975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:07.639184 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.639133    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:07.639184 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.639152    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:07.639184 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.639164    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:07.639332 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.639220    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:15.639202862 +0000 UTC m=+18.157340368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:07.639581 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.639562    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:07.639634 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.639620    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:15.639603209 +0000 UTC m=+18.157740716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:07.957519 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:07.957477    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:07.957758 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:07.957564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:07.957931 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.957903    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:07.958336 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:07.957821    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:08.956156 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:08.955880    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:08.956561 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:08.956261    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:09.956623 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:09.956588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:09.957087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:09.956589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:09.957087 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:09.956734    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:09.957087 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:09.956800    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:10.956388 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:10.956354    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:10.956567 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:10.956505    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:11.956964 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:11.956861    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:11.956964 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:11.956895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:11.957391 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:11.957019    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:11.957457 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:11.957435    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:12.956824 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:12.956785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:12.957040 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:12.956937    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:13.956554 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:13.956519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:13.956554 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:13.956543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:13.956797 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:13.956656    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:13.956848 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:13.956789    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:14.956757 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:14.956726    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:14.957180 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:14.956821    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:15.605746 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:15.605709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:15.606028 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.605842    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:15.606028 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.605914    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.605893618 +0000 UTC m=+34.124031126 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:15.706465 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:15.706423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:15.706648 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:15.706505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:15.706648 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.706589    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:15.706760 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.706652    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:15.706760 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.706666    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.706648707 +0000 UTC m=+34.224786215 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:15.706760 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.706670    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:15.706760 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.706690    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:15.706760 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.706738    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:31.706723183 +0000 UTC m=+34.224860713 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:15.956989 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:15.956886    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:15.957440 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.957055    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:15.957440 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:15.957410    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:15.957571 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:15.957507    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:16.956228 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:16.956195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:16.956432 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:16.956337    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:17.957120 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:17.957086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:17.957551 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:17.957196    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:17.957551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:17.957249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:17.957551 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:17.957334    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:18.956373 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:18.956007    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:18.956373 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:18.956105    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:19.072353 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.072063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6xnjs" event={"ID":"5b9f9f07-7d47-4c98-a206-f1273c62974e","Type":"ContainerStarted","Data":"ec65ae78d3b224af93568eeb73c739dd4bf30d2afff79c6bc0071d2fcdb3000f"}
Sep 09 19:55:19.074634 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.074609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"c1da4a374e41f8bf05b3b98d288162ee99c5603b849235dab932853e59f9d5a6"}
Sep 09 19:55:19.074751 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.074669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"5dcf978c598c89857e94fcbc3ad68dcb286616214e81231fd26741e55f9197cc"}
Sep 09 19:55:19.076320 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.076285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t9j5m" event={"ID":"1f736b1b-7aa0-4509-a18b-c25c0ff131b6","Type":"ContainerStarted","Data":"b718ab04f48a29e523e81f7b912debc0cb7e5aefa522cbc0c28d8825b0f9b69d"}
Sep 09 19:55:19.077737 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.077716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c7sbj" event={"ID":"7b05865d-a410-494c-803b-b6f4f63195fc","Type":"ContainerStarted","Data":"02d6d3da0cd0cf04b7d97e482b96aa235639028d17aba2f070611d8934340ff5"}
Sep 09 19:55:19.079607 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.079555    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" event={"ID":"c620202158ca74d37111884761e47e70","Type":"ContainerStarted","Data":"b162992868f8f981ccfd04df7debe81968ae81a27cc72718ecfc7b8a57b97d8e"}
Sep 09 19:55:19.081101 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.081085    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf" event={"ID":"044e22d3-2279-402a-bb91-e543cd59575f","Type":"ContainerStarted","Data":"6bc63c4a80f08275ea2c39c28f916c5ba852a133cafe8324cadf10c49c7de25b"}
Sep 09 19:55:19.082389 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.082371    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-92f2s" event={"ID":"2a7c2517-8326-4338-ab93-657f631befac","Type":"ContainerStarted","Data":"135fd5b5fbb956fd3c69dcc84a888e8cb8911c8f7a1564abfcfc347ba2df4c06"}
Sep 09 19:55:19.083824 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.083801    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vbmbt" event={"ID":"5731e2df-b062-43b0-9921-5fdf25a9403a","Type":"ContainerStarted","Data":"dd91106609defed1c1c0e7b2633d6b157279d2b5a89c16b076e05afa8ce232b6"}
Sep 09 19:55:19.085586 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.085565    2576 generic.go:356] "Generic (PLEG): container finished" podID="c21d1112-0d32-4c3e-9e79-15322990e4a5" containerID="2e759a51ae2274e45182b06a457f8358dfd38df90b5186f11f2920f4753b544f" exitCode=0
Sep 09 19:55:19.085701 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.085684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerDied","Data":"2e759a51ae2274e45182b06a457f8358dfd38df90b5186f11f2920f4753b544f"}
Sep 09 19:55:19.091121 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.090799    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6xnjs" podStartSLOduration=2.7939350750000003 podStartE2EDuration="21.090786929s" totalImagesPullingTime="18.296851854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.090607786 +0000 UTC m=+21.608745313" watchObservedRunningTime="2026-09-09 19:55:19.090786929 +0000 UTC m=+21.608924491"
Sep 09 19:55:19.091121 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.090873    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-242.ec2.internal" podStartSLOduration=20.090869083 podStartE2EDuration="20.090869083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:02.045594404 +0000 UTC m=+4.563731933" watchObservedRunningTime="2026-09-09 19:55:19.090869083 +0000 UTC m=+21.609006609"
Sep 09 19:55:19.109061 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.109018    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c7sbj" podStartSLOduration=2.7810650519999998 podStartE2EDuration="21.109002697s" totalImagesPullingTime="18.327937645s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.108663024 +0000 UTC m=+21.626800551" watchObservedRunningTime="2026-09-09 19:55:19.109002697 +0000 UTC m=+21.627140223"
Sep 09 19:55:19.149144 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.149092    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vbmbt" podStartSLOduration=2.793419989 podStartE2EDuration="21.149070307s" totalImagesPullingTime="18.355650318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.148635146 +0000 UTC m=+21.666772673" watchObservedRunningTime="2026-09-09 19:55:19.149070307 +0000 UTC m=+21.667207836"
Sep 09 19:55:19.185687 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.185640    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t9j5m" podStartSLOduration=2.883039394 podStartE2EDuration="21.185625615s" totalImagesPullingTime="18.302586221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.165407638 +0000 UTC m=+21.683545185" watchObservedRunningTime="2026-09-09 19:55:19.185625615 +0000 UTC m=+21.703763142"
Sep 09 19:55:19.188840 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.188821    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:55:19.189695 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.189541    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:55:19.201788 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.201736    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-242.ec2.internal" podStartSLOduration=20.2017211 podStartE2EDuration="20.2017211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.201266346 +0000 UTC m=+21.719403874" watchObservedRunningTime="2026-09-09 19:55:19.2017211 +0000 UTC m=+21.719858651"
Sep 09 19:55:19.201984 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.201801    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-92f2s" podStartSLOduration=2.90240313 podStartE2EDuration="21.201796943s" totalImagesPullingTime="18.299393813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:19.185779503 +0000 UTC m=+21.703917019" watchObservedRunningTime="2026-09-09 19:55:19.201796943 +0000 UTC m=+21.719934471"
Sep 09 19:55:19.956814 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.956781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:19.957076 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:19.956781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:19.957076 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:19.956890    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:19.957076 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:19.956996    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:20.092285 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.092253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"b16ad1fce60b1e428151f19b1d38c5297d35260de851a33bec5b5ff83bc5554c"}
Sep 09 19:55:20.092685 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.092342    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"d00e9eb0bc2e34eeec594a423e86a32ad9e29ccdc6b7cf08ced70300637a1151"}
Sep 09 19:55:20.092685 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.092394    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"648d0fc0f91213e73d6cd61481a41b025b72a718fc41e8b36569cd29d81cac66"}
Sep 09 19:55:20.092685 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.092441    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"93abc075d3fce34f44297710abf1eca1e94e7567987dfed417bab1469bbabcb7"}
Sep 09 19:55:20.093920 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.093811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dddng" event={"ID":"8ebec34a-6899-4e62-865a-950c94c88c99","Type":"ContainerStarted","Data":"530aef66829e48c26b2c28623ab85a5806a72c7fb95aa6b4e58e7e99204fef7a"}
Sep 09 19:55:20.094894 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.094876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:55:20.095336 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.095319    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t9j5m"
Sep 09 19:55:20.503790 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.503765    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 19:55:20.936142 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.936027    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T19:55:20.503781105Z","UUID":"038cc4cd-807e-4910-9122-7734a3fabb89","Handler":null,"Name":"","Endpoint":""}
Sep 09 19:55:20.937912 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.937890    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 19:55:20.938050 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.937924    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 19:55:20.956532 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:20.956497    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:20.956677 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:20.956628    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:21.098812 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:21.098768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf" event={"ID":"044e22d3-2279-402a-bb91-e543cd59575f","Type":"ContainerStarted","Data":"e81db88f4eba084586be70acea803d962d523173c4e4b03b77e866fa49105740"}
Sep 09 19:55:21.956173 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:21.956137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:21.956344 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:21.956275    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:21.956416 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:21.956345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:21.956470 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:21.956454    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:22.105074 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:22.105036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"b3e221c050feef70c77ae4e6e708f6bba8f2bf09a8ef0650ec69e62093174794"}
Sep 09 19:55:22.956852 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:22.956814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:22.957073 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:22.956948    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:23.956728 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:23.956493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:23.957327 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:23.956550    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:23.957327 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:23.956837    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:23.957327 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:23.956894    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:24.111031 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.110992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf" event={"ID":"044e22d3-2279-402a-bb91-e543cd59575f","Type":"ContainerStarted","Data":"34c4d5fb20022ee38d6d142e4edb88251cdfff77845657482780aac4b1ded2dc"}
Sep 09 19:55:24.112663 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.112630    2576 generic.go:356] "Generic (PLEG): container finished" podID="c21d1112-0d32-4c3e-9e79-15322990e4a5" containerID="50cccb01899e46fb4b3b3cf48f54c06890e56dc362694ee61fd81b3a4a14b21c" exitCode=0
Sep 09 19:55:24.112779 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.112708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerDied","Data":"50cccb01899e46fb4b3b3cf48f54c06890e56dc362694ee61fd81b3a4a14b21c"}
Sep 09 19:55:24.116263 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.116241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" event={"ID":"b6965512-4260-4621-9640-2e7b38066a97","Type":"ContainerStarted","Data":"38b4bb5c6a2b28436b0ef0285e9dcaf555ebedac7b213fff89fd6729c41b941c"}
Sep 09 19:55:24.116603 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.116559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:24.116766 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.116632    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:24.116766 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.116682    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:24.130604 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.130554    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dddng" podStartSLOduration=11.613992578 podStartE2EDuration="26.130536951s" totalImagesPullingTime="14.516544373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:21.116275289 +0000 UTC m=+23.634412815" watchObservedRunningTime="2026-09-09 19:55:24.130536951 +0000 UTC m=+26.648674480"
Sep 09 19:55:24.131219 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.131182    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vvscf" podStartSLOduration=3.486759418 podStartE2EDuration="26.131173237s" totalImagesPullingTime="22.644413819s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:24.129695206 +0000 UTC m=+26.647832735" watchObservedRunningTime="2026-09-09 19:55:24.131173237 +0000 UTC m=+26.649310763"
Sep 09 19:55:24.132208 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.132190    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:24.132403 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.132389    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:24.173347 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.173303    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df" podStartSLOduration=7.824218595 podStartE2EDuration="26.173289962s" totalImagesPullingTime="18.349071367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:24.172497096 +0000 UTC m=+26.690634623" watchObservedRunningTime="2026-09-09 19:55:24.173289962 +0000 UTC m=+26.691427480"
Sep 09 19:55:24.956166 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:24.956129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:24.956370 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:24.956271    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:25.632502 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:25.632341    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpmgb"]
Sep 09 19:55:25.633007 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:25.632596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:25.633007 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:25.632704    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:25.634681 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:25.634645    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mc9xw"]
Sep 09 19:55:25.634752 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:25.634727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:25.634817 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:25.634799    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:25.635484 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:25.635461    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-spnhz"]
Sep 09 19:55:25.635578 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:25.635557    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:25.635684 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:25.635663    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:26.124263 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:26.124233    2576 generic.go:356] "Generic (PLEG): container finished" podID="c21d1112-0d32-4c3e-9e79-15322990e4a5" containerID="c25b9af13ec876f34355e4fff27c48b60b1768ce3fa0b32c04a1d08c774ad2b1" exitCode=0
Sep 09 19:55:26.124428 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:26.124341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerDied","Data":"c25b9af13ec876f34355e4fff27c48b60b1768ce3fa0b32c04a1d08c774ad2b1"}
Sep 09 19:55:26.955900 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:26.955870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:26.956492 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:26.955993    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:26.956492 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:26.956045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:26.956492 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:26.956127    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:27.957198 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:27.957125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:27.957524 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:27.957241    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:28.131677 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:28.131640    2576 generic.go:356] "Generic (PLEG): container finished" podID="c21d1112-0d32-4c3e-9e79-15322990e4a5" containerID="5cc9b8ab94a2e19c0153086f716fb8c39018aeb266bc37503d171aa3a46d8743" exitCode=0
Sep 09 19:55:28.131886 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:28.131708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerDied","Data":"5cc9b8ab94a2e19c0153086f716fb8c39018aeb266bc37503d171aa3a46d8743"}
Sep 09 19:55:28.956597 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:28.956565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:28.956797 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:28.956605    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:28.956797 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:28.956693    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:28.956906 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:28.956816    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:29.956675 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:29.956491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:29.957151 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:29.956756    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mc9xw" podUID="9d1db42d-aaa6-4d16-92f1-4be2b6aaa988"
Sep 09 19:55:30.956984 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:30.956920    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:30.959123 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:30.959078    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-spnhz" podUID="08532342-baae-47cc-acf7-ad5929ed0c38"
Sep 09 19:55:30.959645 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:30.956991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:30.959795 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:30.959772    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gpmgb" podUID="d36f36ea-ac96-484c-85e5-a327e51afe2e"
Sep 09 19:55:31.623196 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.623157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:31.623356 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.623314    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:31.623430 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.623390    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret podName:9d1db42d-aaa6-4d16-92f1-4be2b6aaa988 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.623368378 +0000 UTC m=+66.141505886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret") pod "global-pull-secret-syncer-mc9xw" (UID: "9d1db42d-aaa6-4d16-92f1-4be2b6aaa988") : object "kube-system"/"original-pull-secret" not registered
Sep 09 19:55:31.724317 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.724280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:31.724485 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.724337    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:31.724485 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.724432    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:31.724485 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.724453    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 19:55:31.724485 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.724469    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 19:55:31.724485 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.724481    2576 projected.go:196] Error preparing data for projected volume kube-api-access-754vp for pod openshift-network-diagnostics/network-check-target-gpmgb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:31.724725 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.724500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs podName:08532342-baae-47cc-acf7-ad5929ed0c38 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.724480061 +0000 UTC m=+66.242617571 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs") pod "network-metrics-daemon-spnhz" (UID: "08532342-baae-47cc-acf7-ad5929ed0c38") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 19:55:31.724725 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:31.724516    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp podName:d36f36ea-ac96-484c-85e5-a327e51afe2e nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.724505718 +0000 UTC m=+66.242643240 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-754vp" (UniqueName: "kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp") pod "network-check-target-gpmgb" (UID: "d36f36ea-ac96-484c-85e5-a327e51afe2e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 19:55:31.725191 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.725172    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-242.ec2.internal" event="NodeReady"
Sep 09 19:55:31.725325 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.725314    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 19:55:31.784714 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.784669    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-xlwl2"]
Sep 09 19:55:31.787278 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.787255    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2"
Sep 09 19:55:31.790505 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.790482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.791139 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.791113    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-zplxk\""
Sep 09 19:55:31.791388 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.791374    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.803294 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.803231    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-xlwl2"]
Sep 09 19:55:31.803984 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.803947    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v4q9w"]
Sep 09 19:55:31.805970 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.805938    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jhd5g"]
Sep 09 19:55:31.806386 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.806059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:31.807875 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.807855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:31.808590 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.808485    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 19:55:31.808590 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.808547    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 19:55:31.808709 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.808691    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5xtg\""
Sep 09 19:55:31.810000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.809966    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gmppz\""
Sep 09 19:55:31.810742 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.810724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 19:55:31.810841 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.810733    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.811152 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.811132    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.819452 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.819430    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v4q9w"]
Sep 09 19:55:31.820220 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.820200    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhd5g"]
Sep 09 19:55:31.861185 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.861151    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-pmctv"]
Sep 09 19:55:31.862997 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.862978    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-8zw8s"]
Sep 09 19:55:31.863126 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.863095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:31.865818 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.864706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:31.866075 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.866058    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 09 19:55:31.867333 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.867152    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 09 19:55:31.867333 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.867203    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.867333 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.867228    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 09 19:55:31.867333 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.867258    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-pmgnr\""
Sep 09 19:55:31.867567 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.867356    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 09 19:55:31.868219 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.868198    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76b586d444-bcdqf"]
Sep 09 19:55:31.868412 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.868391    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.868510 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.868494    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qdgk7\""
Sep 09 19:55:31.870019 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.870000    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"]
Sep 09 19:55:31.870616 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.870590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:31.871870 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.871854    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:31.877614 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.876993    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 09 19:55:31.877614 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.877208    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 19:55:31.877614 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.877463    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 09 19:55:31.877801 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.877626    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 19:55:31.878524 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.878507    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-xdvvq\""
Sep 09 19:55:31.878691 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.878536    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.878758 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.878601    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.878850 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.878828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 19:55:31.879189 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.879170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-8zw8s"]
Sep 09 19:55:31.879476 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.879455    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 09 19:55:31.880239 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.879910    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vlq77\""
Sep 09 19:55:31.880239 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.879984    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-pmctv"]
Sep 09 19:55:31.884406 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.884386    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 19:55:31.890811 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.889816    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b586d444-bcdqf"]
Sep 09 19:55:31.891049 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.890863    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"]
Sep 09 19:55:31.925042 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3172166-9c66-4d90-bd65-34942329005a-tmp-dir\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:31.925224 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925054    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfv65\" (UniqueName: \"kubernetes.io/projected/968b1d70-5306-4f75-9153-9334ea8c468e-kube-api-access-cfv65\") pod \"ingress-canary-jhd5g\" (UID: \"968b1d70-5306-4f75-9153-9334ea8c468e\") " pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:31.925224 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3172166-9c66-4d90-bd65-34942329005a-config-volume\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:31.925224 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:31.925393 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rjk\" (UniqueName: \"kubernetes.io/projected/d3172166-9c66-4d90-bd65-34942329005a-kube-api-access-26rjk\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:31.925393 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925272    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/968b1d70-5306-4f75-9153-9334ea8c468e-cert\") pod \"ingress-canary-jhd5g\" (UID: \"968b1d70-5306-4f75-9153-9334ea8c468e\") " pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:31.925393 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.925319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhkj\" (UniqueName: \"kubernetes.io/projected/c233c968-2f04-4bbe-bb90-0bd77e942e5b-kube-api-access-xnhkj\") pod \"network-check-source-685456655f-xlwl2\" (UID: \"c233c968-2f04-4bbe-bb90-0bd77e942e5b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2"
Sep 09 19:55:31.949665 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.949631    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59dc87fb98-gmrgs"]
Sep 09 19:55:31.952561 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.952538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:31.955152 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955065    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 09 19:55:31.955152 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955071    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 19:55:31.955369 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955309    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 09 19:55:31.955369 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955317    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 19:55:31.955473 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955369    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lvt9g\""
Sep 09 19:55:31.955473 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955320    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 09 19:55:31.955686 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.955669    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 09 19:55:31.956162 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.956150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:55:31.958155 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.958137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:55:31.963252 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:31.963234    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59dc87fb98-gmrgs"]
Sep 09 19:55:32.025993 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.025942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e742f61e-59e8-4a45-b3e5-705965dbf9e4-nginx-conf\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:32.026158 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.026158 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3172166-9c66-4d90-bd65-34942329005a-config-volume\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.026158 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.026308 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rjk\" (UniqueName: \"kubernetes.io/projected/d3172166-9c66-4d90-bd65-34942329005a-kube-api-access-26rjk\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.026308 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/968b1d70-5306-4f75-9153-9334ea8c468e-cert\") pod \"ingress-canary-jhd5g\" (UID: \"968b1d70-5306-4f75-9153-9334ea8c468e\") " pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:32.026308 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.026270    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:32.026450 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.026344    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls podName:d3172166-9c66-4d90-bd65-34942329005a nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.526324168 +0000 UTC m=+35.044461677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls") pod "dns-default-v4q9w" (UID: "d3172166-9c66-4d90-bd65-34942329005a") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:32.026507 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-image-registry-private-configuration\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026507 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/648c619e-c3e9-4564-a431-091fb1c8082e-ca-trust-extracted\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026609 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-trusted-ca\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026609 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-bound-sa-token\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026609 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhkj\" (UniqueName: \"kubernetes.io/projected/c233c968-2f04-4bbe-bb90-0bd77e942e5b-kube-api-access-xnhkj\") pod \"network-check-source-685456655f-xlwl2\" (UID: \"c233c968-2f04-4bbe-bb90-0bd77e942e5b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2"
Sep 09 19:55:32.026609 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c2c780d-970d-4944-874f-722ac1a52827-config\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.026803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c2c780d-970d-4944-874f-722ac1a52827-serving-cert\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.026803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-installation-pull-secrets\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm549\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-kube-api-access-qm549\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.026803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d3172166-9c66-4d90-bd65-34942329005a-config-volume\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.026803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026786    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpmbq\" (UniqueName: \"kubernetes.io/projected/3c2c780d-970d-4944-874f-722ac1a52827-kube-api-access-mpmbq\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-registry-certificates\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026884    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3172166-9c66-4d90-bd65-34942329005a-tmp-dir\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.026985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c2c780d-970d-4944-874f-722ac1a52827-trusted-ca\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.027011    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5thb\" (UniqueName: \"kubernetes.io/projected/d274f779-e7ee-4653-853b-632e5ef8926c-kube-api-access-w5thb\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.027043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfv65\" (UniqueName: \"kubernetes.io/projected/968b1d70-5306-4f75-9153-9334ea8c468e-kube-api-access-cfv65\") pod \"ingress-canary-jhd5g\" (UID: \"968b1d70-5306-4f75-9153-9334ea8c468e\") " pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:32.027087 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.027067    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d274f779-e7ee-4653-853b-632e5ef8926c-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.027405 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.027212    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3172166-9c66-4d90-bd65-34942329005a-tmp-dir\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.030834 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.030810    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/968b1d70-5306-4f75-9153-9334ea8c468e-cert\") pod \"ingress-canary-jhd5g\" (UID: \"968b1d70-5306-4f75-9153-9334ea8c468e\") " pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:32.036159 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.036137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rjk\" (UniqueName: \"kubernetes.io/projected/d3172166-9c66-4d90-bd65-34942329005a-kube-api-access-26rjk\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.036447 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.036414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhkj\" (UniqueName: \"kubernetes.io/projected/c233c968-2f04-4bbe-bb90-0bd77e942e5b-kube-api-access-xnhkj\") pod \"network-check-source-685456655f-xlwl2\" (UID: \"c233c968-2f04-4bbe-bb90-0bd77e942e5b\") " pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2"
Sep 09 19:55:32.036935 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.036913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfv65\" (UniqueName: \"kubernetes.io/projected/968b1d70-5306-4f75-9153-9334ea8c468e-kube-api-access-cfv65\") pod \"ingress-canary-jhd5g\" (UID: \"968b1d70-5306-4f75-9153-9334ea8c468e\") " pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:32.097767 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.097727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2"
Sep 09 19:55:32.124117 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.124086    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhd5g"
Sep 09 19:55:32.127585 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-image-registry-private-configuration\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.127725 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-trusted-ca\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.127725 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c2c780d-970d-4944-874f-722ac1a52827-config\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.127725 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-stats-auth\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.127725 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-registry-certificates\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.127934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c2c780d-970d-4944-874f-722ac1a52827-serving-cert\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.127934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127798    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.127934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-installation-pull-secrets\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.127934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpmbq\" (UniqueName: \"kubernetes.io/projected/3c2c780d-970d-4944-874f-722ac1a52827-kube-api-access-mpmbq\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.127934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127887    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d274f779-e7ee-4653-853b-632e5ef8926c-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.127934 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5thb\" (UniqueName: \"kubernetes.io/projected/d274f779-e7ee-4653-853b-632e5ef8926c-kube-api-access-w5thb\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.128478 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.128444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-registry-certificates\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.128745 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.128719    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c2c780d-970d-4944-874f-722ac1a52827-config\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.129181 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129157    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e742f61e-59e8-4a45-b3e5-705965dbf9e4-nginx-conf\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:32.129268 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.127976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e742f61e-59e8-4a45-b3e5-705965dbf9e4-nginx-conf\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:32.129268 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-default-certificate\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.129422 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.129422 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-bound-sa-token\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.129422 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/648c619e-c3e9-4564-a431-091fb1c8082e-ca-trust-extracted\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.129706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm549\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-kube-api-access-qm549\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.129706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:32.129706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c2c780d-970d-4944-874f-722ac1a52827-trusted-ca\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.129706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt95x\" (UniqueName: \"kubernetes.io/projected/240f7196-4f55-4ab8-9503-877e99f7cf0d-kube-api-access-wt95x\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.129706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.129706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.129676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130026    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130046    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b586d444-bcdqf: secret "image-registry-tls" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130117    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls podName:648c619e-c3e9-4564-a431-091fb1c8082e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.630096737 +0000 UTC m=+35.148234259 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls") pod "image-registry-76b586d444-bcdqf" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e") : secret "image-registry-tls" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.130155    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/648c619e-c3e9-4564-a431-091fb1c8082e-ca-trust-extracted\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130581    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130651    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert podName:e742f61e-59e8-4a45-b3e5-705965dbf9e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.630627144 +0000 UTC m=+35.148764650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8zw8s" (UID: "e742f61e-59e8-4a45-b3e5-705965dbf9e4") : secret "networking-console-plugin-cert" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130706    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.130735    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls podName:d274f779-e7ee-4653-853b-632e5ef8926c nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.630727319 +0000 UTC m=+35.148864824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-t7rqt" (UID: "d274f779-e7ee-4653-853b-632e5ef8926c") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.131278    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/d274f779-e7ee-4653-853b-632e5ef8926c-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.134213 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.131406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c2c780d-970d-4944-874f-722ac1a52827-trusted-ca\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.134686 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.134485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-trusted-ca\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.135779 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.135747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3c2c780d-970d-4944-874f-722ac1a52827-serving-cert\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.136152 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.136111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-image-registry-private-configuration\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.136798 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.136779    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-installation-pull-secrets\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.139133 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.139112    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpmbq\" (UniqueName: \"kubernetes.io/projected/3c2c780d-970d-4944-874f-722ac1a52827-kube-api-access-mpmbq\") pod \"console-operator-85b454874b-pmctv\" (UID: \"3c2c780d-970d-4944-874f-722ac1a52827\") " pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.140108 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.140088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-bound-sa-token\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.140340 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.140307    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm549\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-kube-api-access-qm549\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.141105 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.141090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5thb\" (UniqueName: \"kubernetes.io/projected/d274f779-e7ee-4653-853b-632e5ef8926c-kube-api-access-w5thb\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.179652 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.179333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:32.230786 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.230754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt95x\" (UniqueName: \"kubernetes.io/projected/240f7196-4f55-4ab8-9503-877e99f7cf0d-kube-api-access-wt95x\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.230930 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.230817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.230930 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.230849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-stats-auth\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.230930 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.230918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-default-certificate\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.231102 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.230940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.231348 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.231180    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.731158998 +0000 UTC m=+35.249296506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : configmap references non-existent config key: service-ca.crt
Sep 09 19:55:32.231348 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.231269    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:55:32.231348 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.231325    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:32.731310818 +0000 UTC m=+35.249448336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : secret "router-metrics-certs-default" not found
Sep 09 19:55:32.233746 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.233721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-default-certificate\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.233884 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.233838    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-stats-auth\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.246104 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.246060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt95x\" (UniqueName: \"kubernetes.io/projected/240f7196-4f55-4ab8-9503-877e99f7cf0d-kube-api-access-wt95x\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.423156 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.422978    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-xlwl2"]
Sep 09 19:55:32.428667 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.428642    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhd5g"]
Sep 09 19:55:32.460624 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.460595    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-pmctv"]
Sep 09 19:55:32.464348 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:32.464318    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc233c968_2f04_4bbe_bb90_0bd77e942e5b.slice/crio-be95b4331739f8e608410d8459c4c227076ed27b1389ced1aec3b5d59685dc7a WatchSource:0}: Error finding container be95b4331739f8e608410d8459c4c227076ed27b1389ced1aec3b5d59685dc7a: Status 404 returned error can't find the container with id be95b4331739f8e608410d8459c4c227076ed27b1389ced1aec3b5d59685dc7a
Sep 09 19:55:32.465052 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:32.465019    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968b1d70_5306_4f75_9153_9334ea8c468e.slice/crio-09e321c57fa26d52dc4d3c8dcbc951fdfde144dbe13fa4c627161b58f036faaf WatchSource:0}: Error finding container 09e321c57fa26d52dc4d3c8dcbc951fdfde144dbe13fa4c627161b58f036faaf: Status 404 returned error can't find the container with id 09e321c57fa26d52dc4d3c8dcbc951fdfde144dbe13fa4c627161b58f036faaf
Sep 09 19:55:32.465539 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:32.465518    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2c780d_970d_4944_874f_722ac1a52827.slice/crio-6a937cf41ba4b29cc0b76ea58d30029e04656ad49458ff4a393dc9cd7a95063c WatchSource:0}: Error finding container 6a937cf41ba4b29cc0b76ea58d30029e04656ad49458ff4a393dc9cd7a95063c: Status 404 returned error can't find the container with id 6a937cf41ba4b29cc0b76ea58d30029e04656ad49458ff4a393dc9cd7a95063c
Sep 09 19:55:32.533997 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.533949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:32.534095 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.534060    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:32.534141 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.534125    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls podName:d3172166-9c66-4d90-bd65-34942329005a nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.534109797 +0000 UTC m=+36.052247302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls") pod "dns-default-v4q9w" (UID: "d3172166-9c66-4d90-bd65-34942329005a") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:32.634761 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.634722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:32.634886 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.634785    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:32.634886 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.634846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:32.634886 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634852    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:32.635047 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634912    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls podName:d274f779-e7ee-4653-853b-632e5ef8926c nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.63489686 +0000 UTC m=+36.153034365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-t7rqt" (UID: "d274f779-e7ee-4653-853b-632e5ef8926c") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:32.635047 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634916    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:32.635047 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634930    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b586d444-bcdqf: secret "image-registry-tls" not found
Sep 09 19:55:32.635047 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634931    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:55:32.635047 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634985    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls podName:648c619e-c3e9-4564-a431-091fb1c8082e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.63497296 +0000 UTC m=+36.153110465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls") pod "image-registry-76b586d444-bcdqf" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e") : secret "image-registry-tls" not found
Sep 09 19:55:32.635047 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.634999    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert podName:e742f61e-59e8-4a45-b3e5-705965dbf9e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.63499255 +0000 UTC m=+36.153130055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8zw8s" (UID: "e742f61e-59e8-4a45-b3e5-705965dbf9e4") : secret "networking-console-plugin-cert" not found
Sep 09 19:55:32.735797 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.735720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.735928 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.735838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:32.735928 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.735919    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.735901039 +0000 UTC m=+36.254038544 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : configmap references non-existent config key: service-ca.crt
Sep 09 19:55:32.736028 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.735994    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:55:32.736069 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:32.736046    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:33.73603387 +0000 UTC m=+36.254171388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : secret "router-metrics-certs-default" not found
Sep 09 19:55:32.956250 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.956214    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:55:32.956250 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.956234    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:55:32.958708 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.958687    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:55:32.959389 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.958921    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trbh8\""
Sep 09 19:55:32.959389 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:32.958981    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt99m\""
Sep 09 19:55:33.147070 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.147038    2576 generic.go:356] "Generic (PLEG): container finished" podID="c21d1112-0d32-4c3e-9e79-15322990e4a5" containerID="1008f27894d6f9229f2047b37ba8a8454aef41ed8750033e57ddd6e0190d1bad" exitCode=0
Sep 09 19:55:33.147232 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.147123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerDied","Data":"1008f27894d6f9229f2047b37ba8a8454aef41ed8750033e57ddd6e0190d1bad"}
Sep 09 19:55:33.148502 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.148476    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhd5g" event={"ID":"968b1d70-5306-4f75-9153-9334ea8c468e","Type":"ContainerStarted","Data":"09e321c57fa26d52dc4d3c8dcbc951fdfde144dbe13fa4c627161b58f036faaf"}
Sep 09 19:55:33.149758 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.149733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pmctv" event={"ID":"3c2c780d-970d-4944-874f-722ac1a52827","Type":"ContainerStarted","Data":"6a937cf41ba4b29cc0b76ea58d30029e04656ad49458ff4a393dc9cd7a95063c"}
Sep 09 19:55:33.151115 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.151096    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2" event={"ID":"c233c968-2f04-4bbe-bb90-0bd77e942e5b","Type":"ContainerStarted","Data":"be95b4331739f8e608410d8459c4c227076ed27b1389ced1aec3b5d59685dc7a"}
Sep 09 19:55:33.542590 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.541982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:33.542590 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.542255    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:33.542590 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.542317    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls podName:d3172166-9c66-4d90-bd65-34942329005a nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.54229888 +0000 UTC m=+38.060436403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls") pod "dns-default-v4q9w" (UID: "d3172166-9c66-4d90-bd65-34942329005a") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.643354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.643436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.643500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.643668    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.643684    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b586d444-bcdqf: secret "image-registry-tls" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.643740    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls podName:648c619e-c3e9-4564-a431-091fb1c8082e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.643722615 +0000 UTC m=+38.161860135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls") pod "image-registry-76b586d444-bcdqf" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e") : secret "image-registry-tls" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.644153    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.644202    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert podName:e742f61e-59e8-4a45-b3e5-705965dbf9e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.644187525 +0000 UTC m=+38.162325034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8zw8s" (UID: "e742f61e-59e8-4a45-b3e5-705965dbf9e4") : secret "networking-console-plugin-cert" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.644252    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:33.644317 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.644285    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls podName:d274f779-e7ee-4653-853b-632e5ef8926c nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.644272807 +0000 UTC m=+38.162410326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-t7rqt" (UID: "d274f779-e7ee-4653-853b-632e5ef8926c") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:33.744409 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.744367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:33.744569 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:33.744478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:33.745029 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.744675    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.744654904 +0000 UTC m=+38.262792425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : configmap references non-existent config key: service-ca.crt
Sep 09 19:55:33.745029 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.744834    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:55:33.745029 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:33.744948    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:35.744929945 +0000 UTC m=+38.263067449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : secret "router-metrics-certs-default" not found
Sep 09 19:55:34.157808 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:34.157769    2576 generic.go:356] "Generic (PLEG): container finished" podID="c21d1112-0d32-4c3e-9e79-15322990e4a5" containerID="d7a487033d0718be8a7063232457bfeadc96d278ccd7ba509d43503f6347cea9" exitCode=0
Sep 09 19:55:34.158242 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:34.157861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerDied","Data":"d7a487033d0718be8a7063232457bfeadc96d278ccd7ba509d43503f6347cea9"}
Sep 09 19:55:35.165272 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.165163    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" event={"ID":"c21d1112-0d32-4c3e-9e79-15322990e4a5","Type":"ContainerStarted","Data":"833b2d535ebbb27ae83e569a812671fee3d1264e48e0da2c981b056e31101dd1"}
Sep 09 19:55:35.197494 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.197437    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sc8xs" podStartSLOduration=7.29351713 podStartE2EDuration="37.197420483s" totalImagesPullingTime="29.903903353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:35.197321567 +0000 UTC m=+37.715459096" watchObservedRunningTime="2026-09-09 19:55:35.197420483 +0000 UTC m=+37.715558010"
Sep 09 19:55:35.564613 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.564518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:35.564789 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.564699    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:35.564789 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.564787    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls podName:d3172166-9c66-4d90-bd65-34942329005a nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.564767222 +0000 UTC m=+42.082904727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls") pod "dns-default-v4q9w" (UID: "d3172166-9c66-4d90-bd65-34942329005a") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:35.665149 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.665111    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:35.665288 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.665169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:35.665288 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.665206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:35.665288 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665266    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:55:35.665400 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665291    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:35.665400 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665291    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:35.665400 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665301    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b586d444-bcdqf: secret "image-registry-tls" not found
Sep 09 19:55:35.665400 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665350    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls podName:648c619e-c3e9-4564-a431-091fb1c8082e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.665336487 +0000 UTC m=+42.183473992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls") pod "image-registry-76b586d444-bcdqf" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e") : secret "image-registry-tls" not found
Sep 09 19:55:35.665400 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665364    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls podName:d274f779-e7ee-4653-853b-632e5ef8926c nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.665358224 +0000 UTC m=+42.183495728 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-t7rqt" (UID: "d274f779-e7ee-4653-853b-632e5ef8926c") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:35.665400 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.665373    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert podName:e742f61e-59e8-4a45-b3e5-705965dbf9e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.665368387 +0000 UTC m=+42.183505892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8zw8s" (UID: "e742f61e-59e8-4a45-b3e5-705965dbf9e4") : secret "networking-console-plugin-cert" not found
Sep 09 19:55:35.766158 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.766130    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:35.766259 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:35.766199    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:35.766312 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.766289    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:55:35.766365 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.766354    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.766338057 +0000 UTC m=+42.284475583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : configmap references non-existent config key: service-ca.crt
Sep 09 19:55:35.766407 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:35.766372    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.766366145 +0000 UTC m=+42.284503649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : secret "router-metrics-certs-default" not found
Sep 09 19:55:38.174669 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.174642    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/0.log"
Sep 09 19:55:38.175151 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.174685    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c2c780d-970d-4944-874f-722ac1a52827" containerID="f5bcb2086aa043ddec64032a369704e15698f8d308fcf8ce8e8dcffe8d4ce6b1" exitCode=255
Sep 09 19:55:38.175151 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.174768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pmctv" event={"ID":"3c2c780d-970d-4944-874f-722ac1a52827","Type":"ContainerDied","Data":"f5bcb2086aa043ddec64032a369704e15698f8d308fcf8ce8e8dcffe8d4ce6b1"}
Sep 09 19:55:38.175151 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.175024    2576 scope.go:117] "RemoveContainer" containerID="f5bcb2086aa043ddec64032a369704e15698f8d308fcf8ce8e8dcffe8d4ce6b1"
Sep 09 19:55:38.176483 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.176458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2" event={"ID":"c233c968-2f04-4bbe-bb90-0bd77e942e5b","Type":"ContainerStarted","Data":"97285bf07f2af7316fc9fe5492168f0114fee9583ab7552a31e698e7a403cfa4"}
Sep 09 19:55:38.178059 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.178035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhd5g" event={"ID":"968b1d70-5306-4f75-9153-9334ea8c468e","Type":"ContainerStarted","Data":"f3074a27cccfcfe8e5dcdfefd2504891d5a547b4efe16affcfc5f304b89bd8fa"}
Sep 09 19:55:38.211101 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.211052    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-xlwl2" podStartSLOduration=2.2497814959999998 podStartE2EDuration="7.21103177s" totalImagesPullingTime="4.961250274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:38.209884932 +0000 UTC m=+40.728022459" watchObservedRunningTime="2026-09-09 19:55:38.21103177 +0000 UTC m=+40.729169299"
Sep 09 19:55:38.225176 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.225129    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jhd5g" podStartSLOduration=2.274336517 podStartE2EDuration="7.225109956s" totalImagesPullingTime="4.950773439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:38.224066154 +0000 UTC m=+40.742203682" watchObservedRunningTime="2026-09-09 19:55:38.225109956 +0000 UTC m=+40.743247484"
Sep 09 19:55:38.525512 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.525470    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7mpmf"]
Sep 09 19:55:38.542133 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.542099    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7mpmf"]
Sep 09 19:55:38.542286 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.542237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.544707 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.544686    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 19:55:38.544815 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.544701    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 19:55:38.544815 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.544716    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zhnk7\""
Sep 09 19:55:38.544815 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.544747    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 19:55:38.545000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.544979    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 19:55:38.690854 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.690817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.691070 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.690908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d11d726f-0be5-407f-91e9-3e189e8ea536-crio-socket\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.691070 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.690943    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d11d726f-0be5-407f-91e9-3e189e8ea536-data-volume\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.691070 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.690988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2vjm\" (UniqueName: \"kubernetes.io/projected/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-api-access-n2vjm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.691070 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.691057    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.792840 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.792735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d11d726f-0be5-407f-91e9-3e189e8ea536-crio-socket\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.793142 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.793123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d11d726f-0be5-407f-91e9-3e189e8ea536-data-volume\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.793257 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.793243    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2vjm\" (UniqueName: \"kubernetes.io/projected/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-api-access-n2vjm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.793374 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.793361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.793575 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.793558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.793872 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:38.793854    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:38.794069 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:38.794058    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.294034838 +0000 UTC m=+41.812172359 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:38.794640 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.794616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d11d726f-0be5-407f-91e9-3e189e8ea536-crio-socket\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.795107 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.795087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d11d726f-0be5-407f-91e9-3e189e8ea536-data-volume\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:38.795607 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:38.795584    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:38.795759 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:38.795747    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:39.29572103 +0000 UTC m=+41.813858554 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:38.806342 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:38.806311    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2vjm\" (UniqueName: \"kubernetes.io/projected/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-api-access-n2vjm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:39.181890 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.181859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 19:55:39.182335 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.182262    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/0.log"
Sep 09 19:55:39.182335 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.182305    2576 generic.go:356] "Generic (PLEG): container finished" podID="3c2c780d-970d-4944-874f-722ac1a52827" containerID="14b3e67d5978247eddfb0921a5bcd2fbb15338a998eaa27a690713bf2301aad4" exitCode=255
Sep 09 19:55:39.182446 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.182409    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pmctv" event={"ID":"3c2c780d-970d-4944-874f-722ac1a52827","Type":"ContainerDied","Data":"14b3e67d5978247eddfb0921a5bcd2fbb15338a998eaa27a690713bf2301aad4"}
Sep 09 19:55:39.182517 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.182501    2576 scope.go:117] "RemoveContainer" containerID="f5bcb2086aa043ddec64032a369704e15698f8d308fcf8ce8e8dcffe8d4ce6b1"
Sep 09 19:55:39.182630 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.182616    2576 scope.go:117] "RemoveContainer" containerID="14b3e67d5978247eddfb0921a5bcd2fbb15338a998eaa27a690713bf2301aad4"
Sep 09 19:55:39.182818 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.182797    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pmctv_openshift-console-operator(3c2c780d-970d-4944-874f-722ac1a52827)\"" pod="openshift-console-operator/console-operator-85b454874b-pmctv" podUID="3c2c780d-970d-4944-874f-722ac1a52827"
Sep 09 19:55:39.298421 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.298383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:39.298624 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.298465    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:39.298624 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.298514    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:40.298499179 +0000 UTC m=+42.816636684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:39.298760 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.298634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:39.298811 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.298783    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:39.298872 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.298859    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:40.29883829 +0000 UTC m=+42.816975810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:39.601481 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.601440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:39.601626 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.601597    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 19:55:39.601694 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.601681    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls podName:d3172166-9c66-4d90-bd65-34942329005a nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.601662715 +0000 UTC m=+50.119800224 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls") pod "dns-default-v4q9w" (UID: "d3172166-9c66-4d90-bd65-34942329005a") : secret "dns-default-metrics-tls" not found
Sep 09 19:55:39.702204 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.702171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:39.702375 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.702260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:39.702375 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.702313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:39.702375 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702313    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 19:55:39.702375 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702348    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76b586d444-bcdqf: secret "image-registry-tls" not found
Sep 09 19:55:39.702375 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702355    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:55:39.702584 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702400    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls podName:648c619e-c3e9-4564-a431-091fb1c8082e nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.702382858 +0000 UTC m=+50.220520368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls") pod "image-registry-76b586d444-bcdqf" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e") : secret "image-registry-tls" not found
Sep 09 19:55:39.702584 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702412    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert podName:e742f61e-59e8-4a45-b3e5-705965dbf9e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.702406123 +0000 UTC m=+50.220543627 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8zw8s" (UID: "e742f61e-59e8-4a45-b3e5-705965dbf9e4") : secret "networking-console-plugin-cert" not found
Sep 09 19:55:39.702584 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702413    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:39.702584 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.702466    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls podName:d274f779-e7ee-4653-853b-632e5ef8926c nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.702449286 +0000 UTC m=+50.220586794 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-t7rqt" (UID: "d274f779-e7ee-4653-853b-632e5ef8926c") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:39.803583 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.803543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:39.803741 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.803655    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:39.803741 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.803714    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.803694641 +0000 UTC m=+50.321832145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : configmap references non-existent config key: service-ca.crt
Sep 09 19:55:39.803741 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.803734    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 09 19:55:39.803894 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:39.803784    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:55:47.803772874 +0000 UTC m=+50.321910379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : secret "router-metrics-certs-default" not found
Sep 09 19:55:39.894311 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:39.894243    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c7sbj_7b05865d-a410-494c-803b-b6f4f63195fc/dns-node-resolver/0.log"
Sep 09 19:55:40.186221 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:40.186144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 19:55:40.186561 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:40.186452    2576 scope.go:117] "RemoveContainer" containerID="14b3e67d5978247eddfb0921a5bcd2fbb15338a998eaa27a690713bf2301aad4"
Sep 09 19:55:40.186603 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:40.186586    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pmctv_openshift-console-operator(3c2c780d-970d-4944-874f-722ac1a52827)\"" pod="openshift-console-operator/console-operator-85b454874b-pmctv" podUID="3c2c780d-970d-4944-874f-722ac1a52827"
Sep 09 19:55:40.309182 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:40.309151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:40.309335 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:40.309247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:40.309335 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:40.309288    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:40.309335 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:40.309329    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 19:55:40.309482 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:40.309365    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:42.309349178 +0000 UTC m=+44.827486687 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:40.309482 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:40.309381    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:42.309374188 +0000 UTC m=+44.827511693 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : configmap "kube-rbac-proxy" not found
Sep 09 19:55:41.094135 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:41.094108    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6xnjs_5b9f9f07-7d47-4c98-a206-f1273c62974e/node-ca/0.log"
Sep 09 19:55:41.895221 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:41.895178    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhd5g_968b1d70-5306-4f75-9153-9334ea8c468e/serve-healthcheck-canary/0.log"
Sep 09 19:55:42.179941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:42.179857    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:42.179941 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:42.179912    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:42.180285 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:42.180268    2576 scope.go:117] "RemoveContainer" containerID="14b3e67d5978247eddfb0921a5bcd2fbb15338a998eaa27a690713bf2301aad4"
Sep 09 19:55:42.180434 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:42.180416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-pmctv_openshift-console-operator(3c2c780d-970d-4944-874f-722ac1a52827)\"" pod="openshift-console-operator/console-operator-85b454874b-pmctv" podUID="3c2c780d-970d-4944-874f-722ac1a52827"
Sep 09 19:55:42.324706 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:42.324664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:42.324900 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:42.324770    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:42.324900 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:42.324891    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:42.325042 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:42.324990    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:46.324944693 +0000 UTC m=+48.843082200 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:42.325358 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:42.325332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d11d726f-0be5-407f-91e9-3e189e8ea536-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:46.356108 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:46.356070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:46.356479 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:46.356219    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:46.356479 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:46.356281    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls podName:d11d726f-0be5-407f-91e9-3e189e8ea536 nodeName:}" failed. No retries permitted until 2026-09-09 19:55:54.356264681 +0000 UTC m=+56.874402200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7mpmf" (UID: "d11d726f-0be5-407f-91e9-3e189e8ea536") : secret "insights-runtime-extractor-tls" not found
Sep 09 19:55:47.666326 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.666284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:47.668651 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.668633    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d3172166-9c66-4d90-bd65-34942329005a-metrics-tls\") pod \"dns-default-v4q9w\" (UID: \"d3172166-9c66-4d90-bd65-34942329005a\") " pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:47.717149 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.717118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.767376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.767459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.767497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:47.767586    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:47.767635    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls podName:d274f779-e7ee-4653-853b-632e5ef8926c nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.767622099 +0000 UTC m=+66.285759604 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-t7rqt" (UID: "d274f779-e7ee-4653-853b-632e5ef8926c") : secret "cluster-monitoring-operator-tls" not found
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:47.767769    2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 09 19:55:47.767876 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:47.767828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert podName:e742f61e-59e8-4a45-b3e5-705965dbf9e4 nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.767815395 +0000 UTC m=+66.285952905 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-8zw8s" (UID: "e742f61e-59e8-4a45-b3e5-705965dbf9e4") : secret "networking-console-plugin-cert" not found
Sep 09 19:55:47.770003 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.769980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"image-registry-76b586d444-bcdqf\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") " pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:47.795132 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.795105    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:47.868672 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.868646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:47.868794 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.868722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:47.868889 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:55:47.868869    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle podName:240f7196-4f55-4ab8-9503-877e99f7cf0d nodeName:}" failed. No retries permitted until 2026-09-09 19:56:03.868826425 +0000 UTC m=+66.386963944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle") pod "router-default-59dc87fb98-gmrgs" (UID: "240f7196-4f55-4ab8-9503-877e99f7cf0d") : configmap references non-existent config key: service-ca.crt
Sep 09 19:55:47.870988 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.870970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/240f7196-4f55-4ab8-9503-877e99f7cf0d-metrics-certs\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:55:47.997609 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:47.997535    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v4q9w"]
Sep 09 19:55:48.001722 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:48.001689    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3172166_9c66_4d90_bd65_34942329005a.slice/crio-a49015b72006524af46d2de55a1d1d00ca7f55fdaaeee4c63bd26d6e43cd199d WatchSource:0}: Error finding container a49015b72006524af46d2de55a1d1d00ca7f55fdaaeee4c63bd26d6e43cd199d: Status 404 returned error can't find the container with id a49015b72006524af46d2de55a1d1d00ca7f55fdaaeee4c63bd26d6e43cd199d
Sep 09 19:55:48.076936 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:48.076831    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76b586d444-bcdqf"]
Sep 09 19:55:48.079439 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:48.079418    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648c619e_c3e9_4564_a431_091fb1c8082e.slice/crio-ec37af1f6968d6a84c54af8e41baec4b96d652a746c25a2d13711319b09bdeea WatchSource:0}: Error finding container ec37af1f6968d6a84c54af8e41baec4b96d652a746c25a2d13711319b09bdeea: Status 404 returned error can't find the container with id ec37af1f6968d6a84c54af8e41baec4b96d652a746c25a2d13711319b09bdeea
Sep 09 19:55:48.206504 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:48.206475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b586d444-bcdqf" event={"ID":"648c619e-c3e9-4564-a431-091fb1c8082e","Type":"ContainerStarted","Data":"2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163"}
Sep 09 19:55:48.206627 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:48.206521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b586d444-bcdqf" event={"ID":"648c619e-c3e9-4564-a431-091fb1c8082e","Type":"ContainerStarted","Data":"ec37af1f6968d6a84c54af8e41baec4b96d652a746c25a2d13711319b09bdeea"}
Sep 09 19:55:48.206627 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:48.206556    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:55:48.207705 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:48.207681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v4q9w" event={"ID":"d3172166-9c66-4d90-bd65-34942329005a","Type":"ContainerStarted","Data":"a49015b72006524af46d2de55a1d1d00ca7f55fdaaeee4c63bd26d6e43cd199d"}
Sep 09 19:55:48.230254 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:48.230207    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76b586d444-bcdqf" podStartSLOduration=17.230194452 podStartE2EDuration="17.230194452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:48.229514606 +0000 UTC m=+50.747652134" watchObservedRunningTime="2026-09-09 19:55:48.230194452 +0000 UTC m=+50.748331979"
Sep 09 19:55:51.222621 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:51.222583    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v4q9w" event={"ID":"d3172166-9c66-4d90-bd65-34942329005a","Type":"ContainerStarted","Data":"76473c885d8f7a9e526d7939eaf08187eb0966540d4504e95fb5cdf2033d60f5"}
Sep 09 19:55:51.223043 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:51.222675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v4q9w" event={"ID":"d3172166-9c66-4d90-bd65-34942329005a","Type":"ContainerStarted","Data":"b4cd4aa9b814e66bca42dc5eef1acc3ce511d17b60b2fb8537e7633e4bf9002e"}
Sep 09 19:55:51.223043 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:51.222725    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:55:54.426281 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:54.426223    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:54.428815 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:54.428790    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d11d726f-0be5-407f-91e9-3e189e8ea536-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7mpmf\" (UID: \"d11d726f-0be5-407f-91e9-3e189e8ea536\") " pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:54.451750 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:54.451714    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7mpmf"
Sep 09 19:55:54.719065 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:54.718946    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-v4q9w" podStartSLOduration=21.401444897 podStartE2EDuration="23.718926788s" totalImagesPullingTime="2.317481891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:51.246248876 +0000 UTC m=+53.764386403" watchObservedRunningTime="2026-09-09 19:55:54.718926788 +0000 UTC m=+57.237064315"
Sep 09 19:55:54.720068 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:54.720043    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7mpmf"]
Sep 09 19:55:54.722837 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:55:54.722809    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11d726f_0be5_407f_91e9_3e189e8ea536.slice/crio-8a6a61ff6f883ddfc6202719de8410effd96148c6a4b6fe6aa6bf847e7bc770f WatchSource:0}: Error finding container 8a6a61ff6f883ddfc6202719de8410effd96148c6a4b6fe6aa6bf847e7bc770f: Status 404 returned error can't find the container with id 8a6a61ff6f883ddfc6202719de8410effd96148c6a4b6fe6aa6bf847e7bc770f
Sep 09 19:55:55.238761 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:55.238718    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mpmf" event={"ID":"d11d726f-0be5-407f-91e9-3e189e8ea536","Type":"ContainerStarted","Data":"955673b32d49212f5df74f534571ffcc515d8b89846485e5f9a6d3814baa34b5"}
Sep 09 19:55:55.238936 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:55.238777    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mpmf" event={"ID":"d11d726f-0be5-407f-91e9-3e189e8ea536","Type":"ContainerStarted","Data":"8a6a61ff6f883ddfc6202719de8410effd96148c6a4b6fe6aa6bf847e7bc770f"}
Sep 09 19:55:55.957218 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:55.957137    2576 scope.go:117] "RemoveContainer" containerID="14b3e67d5978247eddfb0921a5bcd2fbb15338a998eaa27a690713bf2301aad4"
Sep 09 19:55:56.142646 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.142616    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dz5df"
Sep 09 19:55:56.244597 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.244472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 19:55:56.244597 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.244581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-pmctv" event={"ID":"3c2c780d-970d-4944-874f-722ac1a52827","Type":"ContainerStarted","Data":"dee3aa2a26b78f576320387bd59f532a72c043567a2427c6c5fea57f2b9744ae"}
Sep 09 19:55:56.244977 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.244943    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:56.247184 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.247140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mpmf" event={"ID":"d11d726f-0be5-407f-91e9-3e189e8ea536","Type":"ContainerStarted","Data":"1192e1a2b89cb7f8a2f7841865e69e54eb6a41fadc472a56fe94c08d3d540eb8"}
Sep 09 19:55:56.264633 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.264583    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-pmctv" podStartSLOduration=20.314234566 podStartE2EDuration="25.264564596s" totalImagesPullingTime="4.95033003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:56.263798497 +0000 UTC m=+58.781936026" watchObservedRunningTime="2026-09-09 19:55:56.264564596 +0000 UTC m=+58.782702125"
Sep 09 19:55:56.584562 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:56.584492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-pmctv"
Sep 09 19:55:58.255870 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:58.255827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7mpmf" event={"ID":"d11d726f-0be5-407f-91e9-3e189e8ea536","Type":"ContainerStarted","Data":"4d6a547e90962703b85d86e7498bc3b01c4a6210e273526e7bd1638c1310bcdc"}
Sep 09 19:55:58.276406 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:55:58.276357    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7mpmf" podStartSLOduration=17.70420683 podStartE2EDuration="20.276338606s" totalImagesPullingTime="2.572131776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:55:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:55:58.275512769 +0000 UTC m=+60.793650295" watchObservedRunningTime="2026-09-09 19:55:58.276338606 +0000 UTC m=+60.794476128"
Sep 09 19:56:01.228551 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.228515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v4q9w"
Sep 09 19:56:01.570740 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.570663    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"]
Sep 09 19:56:01.573117 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.573102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.577500 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.577478    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 19:56:01.577500 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.577493    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-g7sgx\""
Sep 09 19:56:01.577717 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.577538    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 19:56:01.577717 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.577544    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 19:56:01.577717 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.577608    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 19:56:01.581586 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.581556    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"]
Sep 09 19:56:01.584799 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.583351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.584967 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.584924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"]
Sep 09 19:56:01.586237 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.586213    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 19:56:01.601309 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.601282    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"]
Sep 09 19:56:01.661862 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.661831    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76b586d444-bcdqf"]
Sep 09 19:56:01.685834 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.685804    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08e749f4-f19f-4eae-b46d-7a70458f714a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.686000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.685849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrkx\" (UniqueName: \"kubernetes.io/projected/1404e6a3-7e52-4f04-b2a6-8355e991f9a4-kube-api-access-kdrkx\") pod \"managed-serviceaccount-addon-agent-599f94fb58-zm8xs\" (UID: \"1404e6a3-7e52-4f04-b2a6-8355e991f9a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.686000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.685868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgqbp\" (UniqueName: \"kubernetes.io/projected/08e749f4-f19f-4eae-b46d-7a70458f714a-kube-api-access-lgqbp\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.686000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.685906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08e749f4-f19f-4eae-b46d-7a70458f714a-tmp\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.686000 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.685931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1404e6a3-7e52-4f04-b2a6-8355e991f9a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-599f94fb58-zm8xs\" (UID: \"1404e6a3-7e52-4f04-b2a6-8355e991f9a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.692529 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.692491    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-t4sjh"]
Sep 09 19:56:01.694731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.694713    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:01.697239 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.697198    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 09 19:56:01.697447 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.697433    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 09 19:56:01.697556 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.697540    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-lhpjc\""
Sep 09 19:56:01.703865 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.703845    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-588fdcbc55-94fkx"]
Sep 09 19:56:01.705803 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.705788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.713007 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.712988    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-t4sjh"]
Sep 09 19:56:01.722527 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.722503    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588fdcbc55-94fkx"]
Sep 09 19:56:01.787339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08e749f4-f19f-4eae-b46d-7a70458f714a-tmp\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.787339 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dw5t\" (UniqueName: \"kubernetes.io/projected/7d28ce97-a63d-43dd-9eec-05d56df229e6-kube-api-access-9dw5t\") pod \"downloads-85d98c998f-t4sjh\" (UID: \"7d28ce97-a63d-43dd-9eec-05d56df229e6\") " pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:01.787543 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1404e6a3-7e52-4f04-b2a6-8355e991f9a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-599f94fb58-zm8xs\" (UID: \"1404e6a3-7e52-4f04-b2a6-8355e991f9a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.787543 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24dd073f-2ed3-4923-9ced-2016b4113203-image-registry-private-configuration\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787543 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24dd073f-2ed3-4923-9ced-2016b4113203-ca-trust-extracted\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787676 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24dd073f-2ed3-4923-9ced-2016b4113203-installation-pull-secrets\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787676 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-registry-tls\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787773 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787734    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08e749f4-f19f-4eae-b46d-7a70458f714a-tmp\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.787773 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24dd073f-2ed3-4923-9ced-2016b4113203-trusted-ca\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787884 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24dd073f-2ed3-4923-9ced-2016b4113203-registry-certificates\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787884 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-bound-sa-token\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.787884 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787849    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08e749f4-f19f-4eae-b46d-7a70458f714a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.787884 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6s4z\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-kube-api-access-m6s4z\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.788097 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrkx\" (UniqueName: \"kubernetes.io/projected/1404e6a3-7e52-4f04-b2a6-8355e991f9a4-kube-api-access-kdrkx\") pod \"managed-serviceaccount-addon-agent-599f94fb58-zm8xs\" (UID: \"1404e6a3-7e52-4f04-b2a6-8355e991f9a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.788097 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.787977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgqbp\" (UniqueName: \"kubernetes.io/projected/08e749f4-f19f-4eae-b46d-7a70458f714a-kube-api-access-lgqbp\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.790199 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.790174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1404e6a3-7e52-4f04-b2a6-8355e991f9a4-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-599f94fb58-zm8xs\" (UID: \"1404e6a3-7e52-4f04-b2a6-8355e991f9a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.790393 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.790348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/08e749f4-f19f-4eae-b46d-7a70458f714a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.796100 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.796079    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgqbp\" (UniqueName: \"kubernetes.io/projected/08e749f4-f19f-4eae-b46d-7a70458f714a-kube-api-access-lgqbp\") pod \"klusterlet-addon-workmgr-6d77ffc9f9-lffq4\" (UID: \"08e749f4-f19f-4eae-b46d-7a70458f714a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:01.796500 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.796481    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrkx\" (UniqueName: \"kubernetes.io/projected/1404e6a3-7e52-4f04-b2a6-8355e991f9a4-kube-api-access-kdrkx\") pod \"managed-serviceaccount-addon-agent-599f94fb58-zm8xs\" (UID: \"1404e6a3-7e52-4f04-b2a6-8355e991f9a4\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.889191 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"
Sep 09 19:56:01.889396 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dw5t\" (UniqueName: \"kubernetes.io/projected/7d28ce97-a63d-43dd-9eec-05d56df229e6-kube-api-access-9dw5t\") pod \"downloads-85d98c998f-t4sjh\" (UID: \"7d28ce97-a63d-43dd-9eec-05d56df229e6\") " pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:01.889464 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24dd073f-2ed3-4923-9ced-2016b4113203-image-registry-private-configuration\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889464 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24dd073f-2ed3-4923-9ced-2016b4113203-ca-trust-extracted\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889581 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24dd073f-2ed3-4923-9ced-2016b4113203-installation-pull-secrets\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889581 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-registry-tls\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889679 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889583    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24dd073f-2ed3-4923-9ced-2016b4113203-trusted-ca\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889679 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24dd073f-2ed3-4923-9ced-2016b4113203-registry-certificates\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889679 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-bound-sa-token\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889881 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6s4z\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-kube-api-access-m6s4z\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.889881 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.889811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/24dd073f-2ed3-4923-9ced-2016b4113203-ca-trust-extracted\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.890573 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.890539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24dd073f-2ed3-4923-9ced-2016b4113203-trusted-ca\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.890727 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.890706    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/24dd073f-2ed3-4923-9ced-2016b4113203-registry-certificates\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.892422 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.892390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/24dd073f-2ed3-4923-9ced-2016b4113203-image-registry-private-configuration\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.892650 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.892628    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/24dd073f-2ed3-4923-9ced-2016b4113203-installation-pull-secrets\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.892733 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.892717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-registry-tls\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.900752 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.900732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dw5t\" (UniqueName: \"kubernetes.io/projected/7d28ce97-a63d-43dd-9eec-05d56df229e6-kube-api-access-9dw5t\") pod \"downloads-85d98c998f-t4sjh\" (UID: \"7d28ce97-a63d-43dd-9eec-05d56df229e6\") " pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:01.900840 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.900751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-bound-sa-token\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.902060 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.902031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6s4z\" (UniqueName: \"kubernetes.io/projected/24dd073f-2ed3-4923-9ced-2016b4113203-kube-api-access-m6s4z\") pod \"image-registry-588fdcbc55-94fkx\" (UID: \"24dd073f-2ed3-4923-9ced-2016b4113203\") " pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:01.903678 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:01.903656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:02.003111 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.003083    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:02.013820 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.013798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:02.164158 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.164077    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs"]
Sep 09 19:56:02.167384 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:02.167354    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1404e6a3_7e52_4f04_b2a6_8355e991f9a4.slice/crio-2e714a52716dba3c75fc94f27511d674ca484c89824e00448d3a45e3f1432dbb WatchSource:0}: Error finding container 2e714a52716dba3c75fc94f27511d674ca484c89824e00448d3a45e3f1432dbb: Status 404 returned error can't find the container with id 2e714a52716dba3c75fc94f27511d674ca484c89824e00448d3a45e3f1432dbb
Sep 09 19:56:02.186495 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.186463    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"]
Sep 09 19:56:02.188824 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:02.188804    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08e749f4_f19f_4eae_b46d_7a70458f714a.slice/crio-5c0695db0e679e98537e017f0f48a6532d43616ef3a89be122ef01ecb14513e9 WatchSource:0}: Error finding container 5c0695db0e679e98537e017f0f48a6532d43616ef3a89be122ef01ecb14513e9: Status 404 returned error can't find the container with id 5c0695db0e679e98537e017f0f48a6532d43616ef3a89be122ef01ecb14513e9
Sep 09 19:56:02.269222 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.269187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs" event={"ID":"1404e6a3-7e52-4f04-b2a6-8355e991f9a4","Type":"ContainerStarted","Data":"2e714a52716dba3c75fc94f27511d674ca484c89824e00448d3a45e3f1432dbb"}
Sep 09 19:56:02.270552 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.270521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4" event={"ID":"08e749f4-f19f-4eae-b46d-7a70458f714a","Type":"ContainerStarted","Data":"5c0695db0e679e98537e017f0f48a6532d43616ef3a89be122ef01ecb14513e9"}
Sep 09 19:56:02.283443 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.283414    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-t4sjh"]
Sep 09 19:56:02.285907 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:02.285878    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d28ce97_a63d_43dd_9eec_05d56df229e6.slice/crio-4ddf3f6c39e89ac6a3b3b095bbd9efd2b7e887f6000e6276b3cce7865d844cf0 WatchSource:0}: Error finding container 4ddf3f6c39e89ac6a3b3b095bbd9efd2b7e887f6000e6276b3cce7865d844cf0: Status 404 returned error can't find the container with id 4ddf3f6c39e89ac6a3b3b095bbd9efd2b7e887f6000e6276b3cce7865d844cf0
Sep 09 19:56:02.305227 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:02.305176    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588fdcbc55-94fkx"]
Sep 09 19:56:02.307501 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:02.307477    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24dd073f_2ed3_4923_9ced_2016b4113203.slice/crio-db7f9285194a77a373336de7ef859326137c1ba0acdc21246db4ede016a7effd WatchSource:0}: Error finding container db7f9285194a77a373336de7ef859326137c1ba0acdc21246db4ede016a7effd: Status 404 returned error can't find the container with id db7f9285194a77a373336de7ef859326137c1ba0acdc21246db4ede016a7effd
Sep 09 19:56:03.278009 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.277969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx" event={"ID":"24dd073f-2ed3-4923-9ced-2016b4113203","Type":"ContainerStarted","Data":"5ba9d094f18c1ea56b5d435810b527417bd2ed8a74eb09e3201e3ae878e129ad"}
Sep 09 19:56:03.278561 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.278035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx" event={"ID":"24dd073f-2ed3-4923-9ced-2016b4113203","Type":"ContainerStarted","Data":"db7f9285194a77a373336de7ef859326137c1ba0acdc21246db4ede016a7effd"}
Sep 09 19:56:03.278561 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.278492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:03.280313 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.280285    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-t4sjh" event={"ID":"7d28ce97-a63d-43dd-9eec-05d56df229e6","Type":"ContainerStarted","Data":"4ddf3f6c39e89ac6a3b3b095bbd9efd2b7e887f6000e6276b3cce7865d844cf0"}
Sep 09 19:56:03.707418 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.706979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:56:03.709883 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.709782    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 19:56:03.744348 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.744312    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d1db42d-aaa6-4d16-92f1-4be2b6aaa988-original-pull-secret\") pod \"global-pull-secret-syncer-mc9xw\" (UID: \"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988\") " pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:56:03.772753 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.772366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mc9xw"
Sep 09 19:56:03.808800 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.807843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:56:03.808800 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.807902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:56:03.808800 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.807941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:56:03.808800 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.808000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:56:03.811804 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.811584    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 19:56:03.821411 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.821358    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-754vp\" (UniqueName: \"kubernetes.io/projected/d36f36ea-ac96-484c-85e5-a327e51afe2e-kube-api-access-754vp\") pod \"network-check-target-gpmgb\" (UID: \"d36f36ea-ac96-484c-85e5-a327e51afe2e\") " pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:56:03.823659 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.823186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/d274f779-e7ee-4653-853b-632e5ef8926c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-t7rqt\" (UID: \"d274f779-e7ee-4653-853b-632e5ef8926c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:56:03.823659 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.823602    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e742f61e-59e8-4a45-b3e5-705965dbf9e4-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-8zw8s\" (UID: \"e742f61e-59e8-4a45-b3e5-705965dbf9e4\") " pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:56:03.824193 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.824152    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08532342-baae-47cc-acf7-ad5929ed0c38-metrics-certs\") pod \"network-metrics-daemon-spnhz\" (UID: \"08532342-baae-47cc-acf7-ad5929ed0c38\") " pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:56:03.868446 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.868214    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-trbh8\""
Sep 09 19:56:03.873003 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.872804    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xt99m\""
Sep 09 19:56:03.876261 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.875989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:56:03.881010 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.880897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-spnhz"
Sep 09 19:56:03.908788 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.908760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:03.909552 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.909530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/240f7196-4f55-4ab8-9503-877e99f7cf0d-service-ca-bundle\") pod \"router-default-59dc87fb98-gmrgs\" (UID: \"240f7196-4f55-4ab8-9503-877e99f7cf0d\") " pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:03.991488 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.991268    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qdgk7\""
Sep 09 19:56:03.998765 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:03.998625    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s"
Sep 09 19:56:04.003323 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.003137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-vlq77\""
Sep 09 19:56:04.014879 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.011121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"
Sep 09 19:56:04.073133 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.073089    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lvt9g\""
Sep 09 19:56:04.077888 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.075215    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:04.097892 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.097280    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx" podStartSLOduration=3.09726207 podStartE2EDuration="3.09726207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:03.308137904 +0000 UTC m=+65.826275433" watchObservedRunningTime="2026-09-09 19:56:04.09726207 +0000 UTC m=+66.615399625"
Sep 09 19:56:04.099889 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.099793    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mc9xw"]
Sep 09 19:56:04.184400 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.184365    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gpmgb"]
Sep 09 19:56:04.230652 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:04.230493    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-spnhz"]
Sep 09 19:56:05.082130 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:05.081887    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d1db42d_aaa6_4d16_92f1_4be2b6aaa988.slice/crio-1106c4d0df244b0dadfc6f2398625a98a52c20f24b0d73d024c35118d1d2147f WatchSource:0}: Error finding container 1106c4d0df244b0dadfc6f2398625a98a52c20f24b0d73d024c35118d1d2147f: Status 404 returned error can't find the container with id 1106c4d0df244b0dadfc6f2398625a98a52c20f24b0d73d024c35118d1d2147f
Sep 09 19:56:05.293689 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:05.293616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mc9xw" event={"ID":"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988","Type":"ContainerStarted","Data":"1106c4d0df244b0dadfc6f2398625a98a52c20f24b0d73d024c35118d1d2147f"}
Sep 09 19:56:06.018777 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:06.018741    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08532342_baae_47cc_acf7_ad5929ed0c38.slice/crio-a77c0b1633348aa1b3a7663a41f9313370c06f8d4a44ac6f6398323c53219ae1 WatchSource:0}: Error finding container a77c0b1633348aa1b3a7663a41f9313370c06f8d4a44ac6f6398323c53219ae1: Status 404 returned error can't find the container with id a77c0b1633348aa1b3a7663a41f9313370c06f8d4a44ac6f6398323c53219ae1
Sep 09 19:56:06.299189 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:06.299098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-spnhz" event={"ID":"08532342-baae-47cc-acf7-ad5929ed0c38","Type":"ContainerStarted","Data":"a77c0b1633348aa1b3a7663a41f9313370c06f8d4a44ac6f6398323c53219ae1"}
Sep 09 19:56:06.480763 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:06.480733    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd36f36ea_ac96_484c_85e5_a327e51afe2e.slice/crio-58fcebc11f9196d84569311a457195841ed2e7ee604e68b16040ef75c3da8f9b WatchSource:0}: Error finding container 58fcebc11f9196d84569311a457195841ed2e7ee604e68b16040ef75c3da8f9b: Status 404 returned error can't find the container with id 58fcebc11f9196d84569311a457195841ed2e7ee604e68b16040ef75c3da8f9b
Sep 09 19:56:07.304749 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:07.304703    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpmgb" event={"ID":"d36f36ea-ac96-484c-85e5-a327e51afe2e","Type":"ContainerStarted","Data":"58fcebc11f9196d84569311a457195841ed2e7ee604e68b16040ef75c3da8f9b"}
Sep 09 19:56:08.779552 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:08.779028    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-8zw8s"]
Sep 09 19:56:08.794943 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:08.794911    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt"]
Sep 09 19:56:08.797503 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:08.797482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59dc87fb98-gmrgs"]
Sep 09 19:56:09.050042 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:09.049942    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode742f61e_59e8_4a45_b3e5_705965dbf9e4.slice/crio-4e02fcbfb16639ae3f16138227773298a5fa2aa771c246c67cec4e4182fdc0bc WatchSource:0}: Error finding container 4e02fcbfb16639ae3f16138227773298a5fa2aa771c246c67cec4e4182fdc0bc: Status 404 returned error can't find the container with id 4e02fcbfb16639ae3f16138227773298a5fa2aa771c246c67cec4e4182fdc0bc
Sep 09 19:56:09.050877 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:09.050847    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod240f7196_4f55_4ab8_9503_877e99f7cf0d.slice/crio-fa6ac0f0067c2f74a8f488648aaafcc9746222826973dd583b2b221c207e0781 WatchSource:0}: Error finding container fa6ac0f0067c2f74a8f488648aaafcc9746222826973dd583b2b221c207e0781: Status 404 returned error can't find the container with id fa6ac0f0067c2f74a8f488648aaafcc9746222826973dd583b2b221c207e0781
Sep 09 19:56:09.052087 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:09.051613    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd274f779_e7ee_4653_853b_632e5ef8926c.slice/crio-73f1315a4de4250b82b21127767391eb0058aaba29417e1ba46099ca4925850c WatchSource:0}: Error finding container 73f1315a4de4250b82b21127767391eb0058aaba29417e1ba46099ca4925850c: Status 404 returned error can't find the container with id 73f1315a4de4250b82b21127767391eb0058aaba29417e1ba46099ca4925850c
Sep 09 19:56:09.316977 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.316913    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-spnhz" event={"ID":"08532342-baae-47cc-acf7-ad5929ed0c38","Type":"ContainerStarted","Data":"709f23e4a58e7d832d1851d2a90022d3058b989c3bbf023cf2f2d013ca949900"}
Sep 09 19:56:09.317134 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.317014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-spnhz" event={"ID":"08532342-baae-47cc-acf7-ad5929ed0c38","Type":"ContainerStarted","Data":"bd84b8ba5f46cd483eb98b59066f9a6c454eb0866b560497d5816b8f85142ea6"}
Sep 09 19:56:09.318435 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.318378    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt" event={"ID":"d274f779-e7ee-4653-853b-632e5ef8926c","Type":"ContainerStarted","Data":"73f1315a4de4250b82b21127767391eb0058aaba29417e1ba46099ca4925850c"}
Sep 09 19:56:09.320283 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.320229    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s" event={"ID":"e742f61e-59e8-4a45-b3e5-705965dbf9e4","Type":"ContainerStarted","Data":"4e02fcbfb16639ae3f16138227773298a5fa2aa771c246c67cec4e4182fdc0bc"}
Sep 09 19:56:09.322260 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.322191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59dc87fb98-gmrgs" event={"ID":"240f7196-4f55-4ab8-9503-877e99f7cf0d","Type":"ContainerStarted","Data":"88245ace7b11124aca51553b5bf162251af77c39764cfb4bbbbf8e9c9fb175a3"}
Sep 09 19:56:09.322260 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.322226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59dc87fb98-gmrgs" event={"ID":"240f7196-4f55-4ab8-9503-877e99f7cf0d","Type":"ContainerStarted","Data":"fa6ac0f0067c2f74a8f488648aaafcc9746222826973dd583b2b221c207e0781"}
Sep 09 19:56:09.323860 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.323822    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs" event={"ID":"1404e6a3-7e52-4f04-b2a6-8355e991f9a4","Type":"ContainerStarted","Data":"06791a3ebf7c3da69c701472766f17c63f442b6487ff4dbd34da36108bd3c476"}
Sep 09 19:56:09.326365 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.326345    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4" event={"ID":"08e749f4-f19f-4eae-b46d-7a70458f714a","Type":"ContainerStarted","Data":"790ad63957d62ae3d2cba9fcddf69323d2c71c2b018c53b562869288caa327a0"}
Sep 09 19:56:09.326607 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.326568    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:09.328463 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.328443    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4"
Sep 09 19:56:09.329343 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.329322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gpmgb" event={"ID":"d36f36ea-ac96-484c-85e5-a327e51afe2e","Type":"ContainerStarted","Data":"4840b9a57798a3edd2a2391e6f4a1cf5e4b84c73caeea544aac7f0effc838ac9"}
Sep 09 19:56:09.329679 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.329661    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:56:09.338318 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.338139    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-spnhz" podStartSLOduration=68.276289715 podStartE2EDuration="1m11.338125886s" totalImagesPullingTime="3.061836171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:09.336851621 +0000 UTC m=+71.854989147" watchObservedRunningTime="2026-09-09 19:56:09.338125886 +0000 UTC m=+71.856263454"
Sep 09 19:56:09.357609 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.357557    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59dc87fb98-gmrgs" podStartSLOduration=38.357532819 podStartE2EDuration="38.357532819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:09.35589814 +0000 UTC m=+71.874035667" watchObservedRunningTime="2026-09-09 19:56:09.357532819 +0000 UTC m=+71.875670344"
Sep 09 19:56:09.373202 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.373155    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6d77ffc9f9-lffq4" podStartSLOduration=2.047084075 podStartE2EDuration="8.373141686s" totalImagesPullingTime="6.326057611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:09.372562353 +0000 UTC m=+71.890699881" watchObservedRunningTime="2026-09-09 19:56:09.373141686 +0000 UTC m=+71.891279212"
Sep 09 19:56:09.390694 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.390654    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gpmgb" podStartSLOduration=71.390638669 podStartE2EDuration="1m11.390638669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:09.389446245 +0000 UTC m=+71.907583771" watchObservedRunningTime="2026-09-09 19:56:09.390638669 +0000 UTC m=+71.908776197"
Sep 09 19:56:09.410907 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:09.410861    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-599f94fb58-zm8xs" podStartSLOduration=2.0881059730000002 podStartE2EDuration="8.41084219s" totalImagesPullingTime="6.322736217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:09.410418523 +0000 UTC m=+71.928556051" watchObservedRunningTime="2026-09-09 19:56:09.41084219 +0000 UTC m=+71.928979719"
Sep 09 19:56:10.076422 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:10.076347    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:10.079674 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:10.079643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:10.345875 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:10.345774    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:10.346608 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:10.346351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59dc87fb98-gmrgs"
Sep 09 19:56:11.351106 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:11.351020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s" event={"ID":"e742f61e-59e8-4a45-b3e5-705965dbf9e4","Type":"ContainerStarted","Data":"88a704eb907a06000a251d9a88e59b703f8ce91df80a9ab790083a2b1be89d60"}
Sep 09 19:56:11.369590 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:11.368804    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-8zw8s" podStartSLOduration=38.70713227 podStartE2EDuration="40.368782585s" totalImagesPullingTime="1.661650315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:11.367783101 +0000 UTC m=+73.885920642" watchObservedRunningTime="2026-09-09 19:56:11.368782585 +0000 UTC m=+73.886920113"
Sep 09 19:56:11.669150 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:11.668931    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:56:12.357841 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:12.357382    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt" event={"ID":"d274f779-e7ee-4653-853b-632e5ef8926c","Type":"ContainerStarted","Data":"2f98378284c674be02f28b0b08f7d4e0c5c6a73705399eab2e7ee402b8839695"}
Sep 09 19:56:12.376675 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:12.376614    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-t7rqt" podStartSLOduration=38.770897139 podStartE2EDuration="41.37659181s" totalImagesPullingTime="2.605694671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:12.375017187 +0000 UTC m=+74.893154713" watchObservedRunningTime="2026-09-09 19:56:12.37659181 +0000 UTC m=+74.894729338"
Sep 09 19:56:20.673017 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.672877    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rgff6"]
Sep 09 19:56:20.733315 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.733283    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.736253 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.736193    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 19:56:20.736253 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.736244    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 19:56:20.736442 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.736258    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 19:56:20.737066 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.736838    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 19:56:20.737066 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.736880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9mskg\""
Sep 09 19:56:20.857550 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-root\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.857731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-sys\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.857731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857629    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4092ca-5f57-4511-a10b-b85798dc89d5-metrics-client-ca\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.857731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.857731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-textfile\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.857731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857704    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbdw4\" (UniqueName: \"kubernetes.io/projected/5d4092ca-5f57-4511-a10b-b85798dc89d5-kube-api-access-qbdw4\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.857731 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857732    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-tls\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.858179 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-wtmp\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.858179 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.857774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959194 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959194 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-textfile\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959194 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959174    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbdw4\" (UniqueName: \"kubernetes.io/projected/5d4092ca-5f57-4511-a10b-b85798dc89d5-kube-api-access-qbdw4\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959539 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-tls\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959539 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959315    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-wtmp\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959539 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959539 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959511    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-textfile\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959762 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-root\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959762 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-wtmp\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959762 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959609    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-sys\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959762 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959616    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-root\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959762 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4092ca-5f57-4511-a10b-b85798dc89d5-metrics-client-ca\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.959762 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.959656    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5d4092ca-5f57-4511-a10b-b85798dc89d5-sys\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.960091 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.960027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.960144 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.960128    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d4092ca-5f57-4511-a10b-b85798dc89d5-metrics-client-ca\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.962123 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.962100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.962273 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.962238    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5d4092ca-5f57-4511-a10b-b85798dc89d5-node-exporter-tls\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:20.967573 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:20.967548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbdw4\" (UniqueName: \"kubernetes.io/projected/5d4092ca-5f57-4511-a10b-b85798dc89d5-kube-api-access-qbdw4\") pod \"node-exporter-rgff6\" (UID: \"5d4092ca-5f57-4511-a10b-b85798dc89d5\") " pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:21.049258 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:21.049228    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgff6"
Sep 09 19:56:21.723613 ip-10-0-135-242 kubenswrapper[2576]: W0909 19:56:21.723584    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4092ca_5f57_4511_a10b_b85798dc89d5.slice/crio-62f34079bad2fd3c05eb4bcb39485a7ed147b9a8cc7afb9091aa55962fa06df4 WatchSource:0}: Error finding container 62f34079bad2fd3c05eb4bcb39485a7ed147b9a8cc7afb9091aa55962fa06df4: Status 404 returned error can't find the container with id 62f34079bad2fd3c05eb4bcb39485a7ed147b9a8cc7afb9091aa55962fa06df4
Sep 09 19:56:22.400177 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.400139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mc9xw" event={"ID":"9d1db42d-aaa6-4d16-92f1-4be2b6aaa988","Type":"ContainerStarted","Data":"b977b87167116a9d6d816abe098426353b110d2cae8ccbb514fa1e70ad852819"}
Sep 09 19:56:22.402500 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.402471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-t4sjh" event={"ID":"7d28ce97-a63d-43dd-9eec-05d56df229e6","Type":"ContainerStarted","Data":"74348f7c748b69abb31d50096e5c2077145a5b8cfb11e8f53156e90711859ee6"}
Sep 09 19:56:22.402927 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.402903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:22.404240 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.404216    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgff6" event={"ID":"5d4092ca-5f57-4511-a10b-b85798dc89d5","Type":"ContainerStarted","Data":"62f34079bad2fd3c05eb4bcb39485a7ed147b9a8cc7afb9091aa55962fa06df4"}
Sep 09 19:56:22.408769 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.408747    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-t4sjh"
Sep 09 19:56:22.435574 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.435507    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mc9xw" podStartSLOduration=67.797386723 podStartE2EDuration="1m24.435483864s" totalImagesPullingTime="16.638097141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 19:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.435386083 +0000 UTC m=+84.953523620" watchObservedRunningTime="2026-09-09 19:56:22.435483864 +0000 UTC m=+84.953621391"
Sep 09 19:56:22.471513 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:22.471445    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-t4sjh" podStartSLOduration=1.99261408 podStartE2EDuration="21.471425145s" totalImagesPullingTime="19.478811065s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 19:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:22.470160827 +0000 UTC m=+84.988298355" watchObservedRunningTime="2026-09-09 19:56:22.471425145 +0000 UTC m=+84.989562674"
Sep 09 19:56:24.292382 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:24.292351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-588fdcbc55-94fkx"
Sep 09 19:56:24.415394 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:24.415358    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d4092ca-5f57-4511-a10b-b85798dc89d5" containerID="1d81528a4ad719c25c853204321ef6230df1b871a4e0ced7d206211d50751d2d" exitCode=0
Sep 09 19:56:24.416446 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:24.416416    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgff6" event={"ID":"5d4092ca-5f57-4511-a10b-b85798dc89d5","Type":"ContainerDied","Data":"1d81528a4ad719c25c853204321ef6230df1b871a4e0ced7d206211d50751d2d"}
Sep 09 19:56:25.423125 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:25.423083    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgff6" event={"ID":"5d4092ca-5f57-4511-a10b-b85798dc89d5","Type":"ContainerStarted","Data":"3bd313e7d1300112c81e210755912748b2da810cacaa040be307c9975fcdc49d"}
Sep 09 19:56:25.423597 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:25.423187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgff6" event={"ID":"5d4092ca-5f57-4511-a10b-b85798dc89d5","Type":"ContainerStarted","Data":"c063aa3d884c72d0141900b752f391cb06f9aeda0d2d3f713f512b32830b85fb"}
Sep 09 19:56:25.460657 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:25.460596    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rgff6" podStartSLOduration=3.284065686 podStartE2EDuration="5.460579949s" totalImagesPullingTime="2.081705382s" totalInitContainerRuntime="94.808881ms" isStatefulPod=true podCreationTimestamp="2026-09-09 19:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 19:56:25.450039043 +0000 UTC m=+87.968176572" watchObservedRunningTime="2026-09-09 19:56:25.460579949 +0000 UTC m=+87.978717490"
Sep 09 19:56:26.681908 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:26.681863    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76b586d444-bcdqf" podUID="648c619e-c3e9-4564-a431-091fb1c8082e" containerName="registry" containerID="cri-o://2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163" gracePeriod=30
Sep 09 19:56:26.976676 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:26.976647    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:56:27.115852 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.115817    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-installation-pull-secrets\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116065 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.115871    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-registry-certificates\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116065 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.115906    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-image-registry-private-configuration\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116065 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.115932    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116065 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.115989    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-bound-sa-token\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116065 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.116017    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm549\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-kube-api-access-qm549\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116301 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.116074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/648c619e-c3e9-4564-a431-091fb1c8082e-ca-trust-extracted\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116301 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.116098    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-trusted-ca\") pod \"648c619e-c3e9-4564-a431-091fb1c8082e\" (UID: \"648c619e-c3e9-4564-a431-091fb1c8082e\") "
Sep 09 19:56:27.116301 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.116279    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:56:27.116450 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.116367    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-registry-certificates\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.116693 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.116650    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 19:56:27.119217 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.119189    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:27.119497 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.119462    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-kube-api-access-qm549" (OuterVolumeSpecName: "kube-api-access-qm549") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "kube-api-access-qm549". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:27.122395 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.122367    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:56:27.122562 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.122529    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 19:56:27.123993 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.123948    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 19:56:27.127610 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.127583    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/648c619e-c3e9-4564-a431-091fb1c8082e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "648c619e-c3e9-4564-a431-091fb1c8082e" (UID: "648c619e-c3e9-4564-a431-091fb1c8082e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 19:56:27.217790 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217707    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-installation-pull-secrets\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.217790 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217750    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/648c619e-c3e9-4564-a431-091fb1c8082e-image-registry-private-configuration\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.217790 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217767    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-registry-tls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.217790 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217782    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-bound-sa-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.218078 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217796    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm549\" (UniqueName: \"kubernetes.io/projected/648c619e-c3e9-4564-a431-091fb1c8082e-kube-api-access-qm549\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.218078 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217811    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/648c619e-c3e9-4564-a431-091fb1c8082e-ca-trust-extracted\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.218078 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.217824    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/648c619e-c3e9-4564-a431-091fb1c8082e-trusted-ca\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 19:56:27.434795 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.434762    2576 generic.go:356] "Generic (PLEG): container finished" podID="648c619e-c3e9-4564-a431-091fb1c8082e" containerID="2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163" exitCode=0
Sep 09 19:56:27.435002 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.434818    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76b586d444-bcdqf"
Sep 09 19:56:27.435002 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.434846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b586d444-bcdqf" event={"ID":"648c619e-c3e9-4564-a431-091fb1c8082e","Type":"ContainerDied","Data":"2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163"}
Sep 09 19:56:27.435002 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.434917    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76b586d444-bcdqf" event={"ID":"648c619e-c3e9-4564-a431-091fb1c8082e","Type":"ContainerDied","Data":"ec37af1f6968d6a84c54af8e41baec4b96d652a746c25a2d13711319b09bdeea"}
Sep 09 19:56:27.435002 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.434945    2576 scope.go:117] "RemoveContainer" containerID="2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163"
Sep 09 19:56:27.447394 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.447367    2576 scope.go:117] "RemoveContainer" containerID="2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163"
Sep 09 19:56:27.447773 ip-10-0-135-242 kubenswrapper[2576]: E0909 19:56:27.447751    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163\": container with ID starting with 2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163 not found: ID does not exist" containerID="2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163"
Sep 09 19:56:27.447852 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.447783    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163"} err="failed to get container status \"2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163\": rpc error: code = NotFound desc = could not find container \"2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163\": container with ID starting with 2e5bfb574ee8166bd11d7674bb6555e7e20368c506a801528429830d0405f163 not found: ID does not exist"
Sep 09 19:56:27.464627 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.464599    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76b586d444-bcdqf"]
Sep 09 19:56:27.473373 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.473313    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76b586d444-bcdqf"]
Sep 09 19:56:27.965204 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:27.965163    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="648c619e-c3e9-4564-a431-091fb1c8082e" path="/var/lib/kubelet/pods/648c619e-c3e9-4564-a431-091fb1c8082e/volumes"
Sep 09 19:56:41.353860 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:41.353825    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gpmgb"
Sep 09 19:56:44.206299 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:44.206268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59dc87fb98-gmrgs_240f7196-4f55-4ab8-9503-877e99f7cf0d/router/0.log"
Sep 09 19:56:44.232026 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:56:44.231994    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhd5g_968b1d70-5306-4f75-9153-9334ea8c468e/serve-healthcheck-canary/0.log"
Sep 09 19:59:57.865376 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:59:57.865349    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 19:59:57.869931 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:59:57.869912    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 19:59:57.874743 ip-10-0-135-242 kubenswrapper[2576]: I0909 19:59:57.874728    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:00:34.311948 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.311909    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"]
Sep 09 20:00:34.313876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.312276    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="648c619e-c3e9-4564-a431-091fb1c8082e" containerName="registry"
Sep 09 20:00:34.313876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.312293    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="648c619e-c3e9-4564-a431-091fb1c8082e" containerName="registry"
Sep 09 20:00:34.313876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.312373    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="648c619e-c3e9-4564-a431-091fb1c8082e" containerName="registry"
Sep 09 20:00:34.314308 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.314290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.316835 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.316814    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:00:34.316931 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.316915    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:00:34.317770 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.317755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rlsmd\""
Sep 09 20:00:34.323690 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.323660    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"]
Sep 09 20:00:34.392166 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.392138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.392336 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.392208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r99q\" (UniqueName: \"kubernetes.io/projected/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-kube-api-access-5r99q\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.392336 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.392276    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.492896 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.492858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r99q\" (UniqueName: \"kubernetes.io/projected/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-kube-api-access-5r99q\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.493101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.492902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.493101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.492925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.493268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.493248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.493310 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.493277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.501562 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.501544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r99q\" (UniqueName: \"kubernetes.io/projected/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-kube-api-access-5r99q\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.623899 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.623866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:34.896092 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.896005    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"]
Sep 09 20:00:34.899797 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:00:34.899772    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272a2cc6_1d2a_4f24_8b69_bb718bb7bca5.slice/crio-7fe30f605e0ca2980385e5368ad09df08e004b8ba939ea46a12a1c241e5fa2e5 WatchSource:0}: Error finding container 7fe30f605e0ca2980385e5368ad09df08e004b8ba939ea46a12a1c241e5fa2e5: Status 404 returned error can't find the container with id 7fe30f605e0ca2980385e5368ad09df08e004b8ba939ea46a12a1c241e5fa2e5
Sep 09 20:00:34.901849 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:34.901833    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:00:35.277882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:35.277849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp" event={"ID":"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5","Type":"ContainerStarted","Data":"7fe30f605e0ca2980385e5368ad09df08e004b8ba939ea46a12a1c241e5fa2e5"}
Sep 09 20:00:42.305405 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:42.305360    2576 generic.go:356] "Generic (PLEG): container finished" podID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerID="0e8710b9a5ea178afffa7d7d0f497ca59e91d85e8bd47dfb4495474e02ede0a4" exitCode=0
Sep 09 20:00:42.305405 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:42.305405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp" event={"ID":"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5","Type":"ContainerDied","Data":"0e8710b9a5ea178afffa7d7d0f497ca59e91d85e8bd47dfb4495474e02ede0a4"}
Sep 09 20:00:45.318155 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:45.318072    2576 generic.go:356] "Generic (PLEG): container finished" podID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerID="fdad0d5932e8d15566c9d7457365f4b289c9760600d5b0398ca0f9b7cbf58543" exitCode=0
Sep 09 20:00:45.318155 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:45.318140    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp" event={"ID":"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5","Type":"ContainerDied","Data":"fdad0d5932e8d15566c9d7457365f4b289c9760600d5b0398ca0f9b7cbf58543"}
Sep 09 20:00:54.356874 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:54.356842    2576 generic.go:356] "Generic (PLEG): container finished" podID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerID="6e034dc32035529d65ed806afa975fad2d8511c6d7aa71f112d2077c9fefe678" exitCode=0
Sep 09 20:00:54.357254 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:54.356899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp" event={"ID":"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5","Type":"ContainerDied","Data":"6e034dc32035529d65ed806afa975fad2d8511c6d7aa71f112d2077c9fefe678"}
Sep 09 20:00:55.479318 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.479294    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:00:55.566426 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.566396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-bundle\") pod \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") "
Sep 09 20:00:55.566589 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.566443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r99q\" (UniqueName: \"kubernetes.io/projected/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-kube-api-access-5r99q\") pod \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") "
Sep 09 20:00:55.566589 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.566475    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-util\") pod \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\" (UID: \"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5\") "
Sep 09 20:00:55.567113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.567085    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-bundle" (OuterVolumeSpecName: "bundle") pod "272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" (UID: "272a2cc6-1d2a-4f24-8b69-bb718bb7bca5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:00:55.568649 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.568625    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-kube-api-access-5r99q" (OuterVolumeSpecName: "kube-api-access-5r99q") pod "272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" (UID: "272a2cc6-1d2a-4f24-8b69-bb718bb7bca5"). InnerVolumeSpecName "kube-api-access-5r99q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:00:55.570625 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.570594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-util" (OuterVolumeSpecName: "util") pod "272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" (UID: "272a2cc6-1d2a-4f24-8b69-bb718bb7bca5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:00:55.667341 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.667264    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:00:55.667341 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.667289    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:00:55.667341 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:55.667300    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r99q\" (UniqueName: \"kubernetes.io/projected/272a2cc6-1d2a-4f24-8b69-bb718bb7bca5-kube-api-access-5r99q\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:00:56.366008 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:56.365974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp" event={"ID":"272a2cc6-1d2a-4f24-8b69-bb718bb7bca5","Type":"ContainerDied","Data":"7fe30f605e0ca2980385e5368ad09df08e004b8ba939ea46a12a1c241e5fa2e5"}
Sep 09 20:00:56.366180 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:56.366022    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe30f605e0ca2980385e5368ad09df08e004b8ba939ea46a12a1c241e5fa2e5"
Sep 09 20:00:56.366180 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:00:56.366033    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a872bp"
Sep 09 20:01:02.836673 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836639    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"]
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836878    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="pull"
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836889    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="pull"
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836899    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="util"
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836904    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="util"
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836911    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="extract"
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836916    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="extract"
Sep 09 20:01:02.837057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.836985    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="272a2cc6-1d2a-4f24-8b69-bb718bb7bca5" containerName="extract"
Sep 09 20:01:02.839185 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.839169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:02.843232 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:02.843201    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kedaorg-certs\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" type="*v1.Secret"
Sep 09 20:01:02.843232 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:02.843198    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 09 20:01:02.843417 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:02.843204    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" podUID="a24b8a23-03b7-497b-9cd4-b77d8cb8ceef" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:02.843417 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:02.843252    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 09 20:01:02.843417 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:02.843263    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"custom-metrics-autoscaler-operator-dockercfg-6vk4k\" is forbidden: User \"system:node:ip-10-0-135-242.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-135-242.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-6vk4k\"" type="*v1.Secret"
Sep 09 20:01:02.862138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.862106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"]
Sep 09 20:01:02.922367 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.922328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8sn7\" (UniqueName: \"kubernetes.io/projected/a24b8a23-03b7-497b-9cd4-b77d8cb8ceef-kube-api-access-x8sn7\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" (UID: \"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:02.922367 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:02.922376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a24b8a23-03b7-497b-9cd4-b77d8cb8ceef-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" (UID: \"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:03.023365 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.023332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8sn7\" (UniqueName: \"kubernetes.io/projected/a24b8a23-03b7-497b-9cd4-b77d8cb8ceef-kube-api-access-x8sn7\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" (UID: \"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:03.023528 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.023373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a24b8a23-03b7-497b-9cd4-b77d8cb8ceef-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" (UID: \"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:03.786908 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.786868    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-6vk4k\""
Sep 09 20:01:03.823852 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.823816    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 09 20:01:03.866641 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.866610    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 09 20:01:03.876149 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.876122    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8sn7\" (UniqueName: \"kubernetes.io/projected/a24b8a23-03b7-497b-9cd4-b77d8cb8ceef-kube-api-access-x8sn7\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" (UID: \"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:03.972199 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.972169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 09 20:01:03.976380 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:03.976361    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/a24b8a23-03b7-497b-9cd4-b77d8cb8ceef-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-svzpz\" (UID: \"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:04.049185 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:04.049100    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:04.322452 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:04.322375    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"]
Sep 09 20:01:04.326016 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:01:04.325989    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24b8a23_03b7_497b_9cd4_b77d8cb8ceef.slice/crio-4aeac3330524439ff1af9467740cc869c8e91c0706721ace080bfd902b6483c2 WatchSource:0}: Error finding container 4aeac3330524439ff1af9467740cc869c8e91c0706721ace080bfd902b6483c2: Status 404 returned error can't find the container with id 4aeac3330524439ff1af9467740cc869c8e91c0706721ace080bfd902b6483c2
Sep 09 20:01:04.395673 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:04.395640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz" event={"ID":"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef","Type":"ContainerStarted","Data":"4aeac3330524439ff1af9467740cc869c8e91c0706721ace080bfd902b6483c2"}
Sep 09 20:01:08.412870 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.412834    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz" event={"ID":"a24b8a23-03b7-497b-9cd4-b77d8cb8ceef","Type":"ContainerStarted","Data":"dad3015569fdd1a8f41865216b0dde4492f6e9ddd83e56c302d605eec50fef37"}
Sep 09 20:01:08.413306 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.413081    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:01:08.440250 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.440045    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz" podStartSLOduration=3.315163092 podStartE2EDuration="6.440026976s" totalImagesPullingTime="3.124863884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:08.439693731 +0000 UTC m=+370.957831331" watchObservedRunningTime="2026-09-09 20:01:08.440026976 +0000 UTC m=+370.958164505"
Sep 09 20:01:08.484662 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.484624    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"]
Sep 09 20:01:08.487268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.487248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.490793 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.490771    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 09 20:01:08.490934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.490815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-zbjxt\""
Sep 09 20:01:08.490934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.490810    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 09 20:01:08.499356 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.499328    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"]
Sep 09 20:01:08.671164 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.671079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.671164 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.671140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5v2w\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-kube-api-access-m5v2w\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.671404 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.671175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.772463 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.772428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.772631 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.772476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5v2w\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-kube-api-access-m5v2w\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.772631 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.772496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.772631 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:08.772611    2576 secret.go:281] references non-existent secret key: tls.crt
Sep 09 20:01:08.772631 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:08.772625    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 09 20:01:08.772774 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:08.772642    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw: references non-existent secret key: tls.crt
Sep 09 20:01:08.772774 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:01:08.772693    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-certificates podName:b096b89d-9b72-4a1d-b40a-69ebdd8e0a34 nodeName:}" failed. No retries permitted until 2026-09-09 20:01:09.272676667 +0000 UTC m=+371.790814190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-certificates") pod "keda-metrics-apiserver-77857b7b98-77qjw" (UID: "b096b89d-9b72-4a1d-b40a-69ebdd8e0a34") : references non-existent secret key: tls.crt
Sep 09 20:01:08.772943 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.772886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:08.781705 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:08.781682    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5v2w\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-kube-api-access-m5v2w\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:09.277515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:09.277474    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:09.280259 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:09.280220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b096b89d-9b72-4a1d-b40a-69ebdd8e0a34-certificates\") pod \"keda-metrics-apiserver-77857b7b98-77qjw\" (UID: \"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:09.397691 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:09.397658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:09.679419 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:09.679393    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"]
Sep 09 20:01:09.681014 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:01:09.680923    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096b89d_9b72_4a1d_b40a_69ebdd8e0a34.slice/crio-0e6b645f7edfdbcbd2fa1b601d28a9f33ec1febf66b592ea5f4a1fef61ac1afb WatchSource:0}: Error finding container 0e6b645f7edfdbcbd2fa1b601d28a9f33ec1febf66b592ea5f4a1fef61ac1afb: Status 404 returned error can't find the container with id 0e6b645f7edfdbcbd2fa1b601d28a9f33ec1febf66b592ea5f4a1fef61ac1afb
Sep 09 20:01:10.422628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:10.422588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw" event={"ID":"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34","Type":"ContainerStarted","Data":"0e6b645f7edfdbcbd2fa1b601d28a9f33ec1febf66b592ea5f4a1fef61ac1afb"}
Sep 09 20:01:14.441268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:14.441236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw" event={"ID":"b096b89d-9b72-4a1d-b40a-69ebdd8e0a34","Type":"ContainerStarted","Data":"cfc9ff180605fe516450916647c38ffa4e851296b3ec90c7e0f3015d3f3ff113"}
Sep 09 20:01:14.441674 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:14.441380    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:14.463308 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:14.462351    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw" podStartSLOduration=2.250075712 podStartE2EDuration="6.46232611s" totalImagesPullingTime="4.212250398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:01:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:01:14.459268876 +0000 UTC m=+376.977406404" watchObservedRunningTime="2026-09-09 20:01:14.46232611 +0000 UTC m=+376.980463639"
Sep 09 20:01:25.450147 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:25.450113    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-77qjw"
Sep 09 20:01:29.420470 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:01:29.420440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-svzpz"
Sep 09 20:02:00.230087 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.230049    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"]
Sep 09 20:02:00.232303 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.232286    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.235042 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.235015    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:02:00.235141 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.235052    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:02:00.235141 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.235084    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rlsmd\""
Sep 09 20:02:00.243561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.243540    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"]
Sep 09 20:02:00.378996 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.378938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4rmw\" (UniqueName: \"kubernetes.io/projected/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-kube-api-access-c4rmw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.379173 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.379003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.379173 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.379046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.479835 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.479800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4rmw\" (UniqueName: \"kubernetes.io/projected/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-kube-api-access-c4rmw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.479835 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.479840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.480070 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.479986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.480268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.480225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.480309 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.480272    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.489668 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.489642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4rmw\" (UniqueName: \"kubernetes.io/projected/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-kube-api-access-c4rmw\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.541606 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.541573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:00.812860 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:00.812810    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"]
Sep 09 20:02:00.815193 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:02:00.815165    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d115f40_d07e_4a2a_bb4c_c2da9f33827e.slice/crio-90e21e354989a80bf08c3064b4a466c91aefed9fa8e1e1adac6babd187b2dc87 WatchSource:0}: Error finding container 90e21e354989a80bf08c3064b4a466c91aefed9fa8e1e1adac6babd187b2dc87: Status 404 returned error can't find the container with id 90e21e354989a80bf08c3064b4a466c91aefed9fa8e1e1adac6babd187b2dc87
Sep 09 20:02:01.621849 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:01.621814    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerID="f6ec0f4f3a6f9dcd9516dcafacc920ca785caf96c6fb67fe46005ee8109c3de4" exitCode=0
Sep 09 20:02:01.622235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:01.621879    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r" event={"ID":"7d115f40-d07e-4a2a-bb4c-c2da9f33827e","Type":"ContainerDied","Data":"f6ec0f4f3a6f9dcd9516dcafacc920ca785caf96c6fb67fe46005ee8109c3de4"}
Sep 09 20:02:01.622235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:01.621916    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r" event={"ID":"7d115f40-d07e-4a2a-bb4c-c2da9f33827e","Type":"ContainerStarted","Data":"90e21e354989a80bf08c3064b4a466c91aefed9fa8e1e1adac6babd187b2dc87"}
Sep 09 20:02:04.635973 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:04.635917    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerID="160a06e1df8c5ad9b572709364e2dfac79e8b5515842d799d1dcf1fc66488a0a" exitCode=0
Sep 09 20:02:04.636343 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:04.635990    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r" event={"ID":"7d115f40-d07e-4a2a-bb4c-c2da9f33827e","Type":"ContainerDied","Data":"160a06e1df8c5ad9b572709364e2dfac79e8b5515842d799d1dcf1fc66488a0a"}
Sep 09 20:02:05.641561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:05.641530    2576 generic.go:356] "Generic (PLEG): container finished" podID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerID="d09590488a5b1bc648161a0eb5d56edc24e2cbe8a6546b84cd94740fa3628fb3" exitCode=0
Sep 09 20:02:05.642033 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:05.641613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r" event={"ID":"7d115f40-d07e-4a2a-bb4c-c2da9f33827e","Type":"ContainerDied","Data":"d09590488a5b1bc648161a0eb5d56edc24e2cbe8a6546b84cd94740fa3628fb3"}
Sep 09 20:02:06.767811 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.767786    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:06.830165 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.830132    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-bundle\") pod \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") "
Sep 09 20:02:06.830334 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.830181    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-util\") pod \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") "
Sep 09 20:02:06.830334 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.830224    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4rmw\" (UniqueName: \"kubernetes.io/projected/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-kube-api-access-c4rmw\") pod \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\" (UID: \"7d115f40-d07e-4a2a-bb4c-c2da9f33827e\") "
Sep 09 20:02:06.830812 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.830776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-bundle" (OuterVolumeSpecName: "bundle") pod "7d115f40-d07e-4a2a-bb4c-c2da9f33827e" (UID: "7d115f40-d07e-4a2a-bb4c-c2da9f33827e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:02:06.832398 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.832370    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-kube-api-access-c4rmw" (OuterVolumeSpecName: "kube-api-access-c4rmw") pod "7d115f40-d07e-4a2a-bb4c-c2da9f33827e" (UID: "7d115f40-d07e-4a2a-bb4c-c2da9f33827e"). InnerVolumeSpecName "kube-api-access-c4rmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:02:06.834606 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.834586    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-util" (OuterVolumeSpecName: "util") pod "7d115f40-d07e-4a2a-bb4c-c2da9f33827e" (UID: "7d115f40-d07e-4a2a-bb4c-c2da9f33827e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:02:06.930926 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.930839    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:02:06.930926 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.930868    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:02:06.930926 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:06.930877    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4rmw\" (UniqueName: \"kubernetes.io/projected/7d115f40-d07e-4a2a-bb4c-c2da9f33827e-kube-api-access-c4rmw\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:02:07.652149 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:07.652112    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r" event={"ID":"7d115f40-d07e-4a2a-bb4c-c2da9f33827e","Type":"ContainerDied","Data":"90e21e354989a80bf08c3064b4a466c91aefed9fa8e1e1adac6babd187b2dc87"}
Sep 09 20:02:07.652320 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:07.652158    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540glt9r"
Sep 09 20:02:07.652320 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:07.652167    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90e21e354989a80bf08c3064b4a466c91aefed9fa8e1e1adac6babd187b2dc87"
Sep 09 20:02:22.649636 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649555    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"]
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649812    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="extract"
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649822    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="extract"
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649836    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="util"
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649841    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="util"
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649849    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="pull"
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649854    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="pull"
Sep 09 20:02:22.650007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.649903    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d115f40-d07e-4a2a-bb4c-c2da9f33827e" containerName="extract"
Sep 09 20:02:22.652026 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.652011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.654745 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.654715    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:02:22.655132 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.655118    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rlsmd\""
Sep 09 20:02:22.655708 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.655695    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:02:22.664027 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.664002    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"]
Sep 09 20:02:22.753385 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.753350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.753587 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.753433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvzc2\" (UniqueName: \"kubernetes.io/projected/97bea1a2-d246-45e9-adc7-bcd634613d00-kube-api-access-wvzc2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.753587 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.753466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.854201 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.854153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvzc2\" (UniqueName: \"kubernetes.io/projected/97bea1a2-d246-45e9-adc7-bcd634613d00-kube-api-access-wvzc2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.854201 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.854202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.854457 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.854238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.854643 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.854621    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.854690 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.854641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.871008 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.870973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvzc2\" (UniqueName: \"kubernetes.io/projected/97bea1a2-d246-45e9-adc7-bcd634613d00-kube-api-access-wvzc2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:22.961419 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:22.961335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:23.261911 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:23.261882    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"]
Sep 09 20:02:23.262842 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:02:23.262815    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97bea1a2_d246_45e9_adc7_bcd634613d00.slice/crio-8aa35e13fb805c4fa7fea71d897ad0e982fb4e2e6f79b7eaa218558dd28c53c7 WatchSource:0}: Error finding container 8aa35e13fb805c4fa7fea71d897ad0e982fb4e2e6f79b7eaa218558dd28c53c7: Status 404 returned error can't find the container with id 8aa35e13fb805c4fa7fea71d897ad0e982fb4e2e6f79b7eaa218558dd28c53c7
Sep 09 20:02:23.717715 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:23.717679    2576 generic.go:356] "Generic (PLEG): container finished" podID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerID="e6408e380af1049ef39629c4bfec4fe8d731059e38950c48bbfa107c07cf8c01" exitCode=0
Sep 09 20:02:23.718162 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:23.717775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf" event={"ID":"97bea1a2-d246-45e9-adc7-bcd634613d00","Type":"ContainerDied","Data":"e6408e380af1049ef39629c4bfec4fe8d731059e38950c48bbfa107c07cf8c01"}
Sep 09 20:02:23.718162 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:23.717876    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf" event={"ID":"97bea1a2-d246-45e9-adc7-bcd634613d00","Type":"ContainerStarted","Data":"8aa35e13fb805c4fa7fea71d897ad0e982fb4e2e6f79b7eaa218558dd28c53c7"}
Sep 09 20:02:26.731687 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:26.731648    2576 generic.go:356] "Generic (PLEG): container finished" podID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerID="0e390946266d829e837b25cef25fdcc29128c459681a02fdabdf638adc505fed" exitCode=0
Sep 09 20:02:26.732073 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:26.731724    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf" event={"ID":"97bea1a2-d246-45e9-adc7-bcd634613d00","Type":"ContainerDied","Data":"0e390946266d829e837b25cef25fdcc29128c459681a02fdabdf638adc505fed"}
Sep 09 20:02:27.738072 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:27.738033    2576 generic.go:356] "Generic (PLEG): container finished" podID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerID="030fe285e1be864ad0e86d09d849a2ccfb6e985bc248193c141f35c812b6c114" exitCode=0
Sep 09 20:02:27.738446 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:27.738086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf" event={"ID":"97bea1a2-d246-45e9-adc7-bcd634613d00","Type":"ContainerDied","Data":"030fe285e1be864ad0e86d09d849a2ccfb6e985bc248193c141f35c812b6c114"}
Sep 09 20:02:28.876134 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:28.876104    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:29.006085 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.005988    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-util\") pod \"97bea1a2-d246-45e9-adc7-bcd634613d00\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") "
Sep 09 20:02:29.006085 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.006045    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvzc2\" (UniqueName: \"kubernetes.io/projected/97bea1a2-d246-45e9-adc7-bcd634613d00-kube-api-access-wvzc2\") pod \"97bea1a2-d246-45e9-adc7-bcd634613d00\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") "
Sep 09 20:02:29.006085 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.006084    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-bundle\") pod \"97bea1a2-d246-45e9-adc7-bcd634613d00\" (UID: \"97bea1a2-d246-45e9-adc7-bcd634613d00\") "
Sep 09 20:02:29.006540 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.006504    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-bundle" (OuterVolumeSpecName: "bundle") pod "97bea1a2-d246-45e9-adc7-bcd634613d00" (UID: "97bea1a2-d246-45e9-adc7-bcd634613d00"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:02:29.008231 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.008209    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97bea1a2-d246-45e9-adc7-bcd634613d00-kube-api-access-wvzc2" (OuterVolumeSpecName: "kube-api-access-wvzc2") pod "97bea1a2-d246-45e9-adc7-bcd634613d00" (UID: "97bea1a2-d246-45e9-adc7-bcd634613d00"). InnerVolumeSpecName "kube-api-access-wvzc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:02:29.087434 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.087396    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-util" (OuterVolumeSpecName: "util") pod "97bea1a2-d246-45e9-adc7-bcd634613d00" (UID: "97bea1a2-d246-45e9-adc7-bcd634613d00"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:02:29.107348 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.107322    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:02:29.107348 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.107345    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/97bea1a2-d246-45e9-adc7-bcd634613d00-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:02:29.107468 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.107354    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvzc2\" (UniqueName: \"kubernetes.io/projected/97bea1a2-d246-45e9-adc7-bcd634613d00-kube-api-access-wvzc2\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:02:29.748904 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.748871    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf" event={"ID":"97bea1a2-d246-45e9-adc7-bcd634613d00","Type":"ContainerDied","Data":"8aa35e13fb805c4fa7fea71d897ad0e982fb4e2e6f79b7eaa218558dd28c53c7"}
Sep 09 20:02:29.749100 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.748921    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa35e13fb805c4fa7fea71d897ad0e982fb4e2e6f79b7eaa218558dd28c53c7"
Sep 09 20:02:29.749100 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:29.748878    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjp4wf"
Sep 09 20:02:31.799636 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799602    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sttnt"]
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799866    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="util"
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799876    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="util"
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799885    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="pull"
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799891    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="pull"
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799897    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="extract"
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799902    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="extract"
Sep 09 20:02:31.800043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.799943    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="97bea1a2-d246-45e9-adc7-bcd634613d00" containerName="extract"
Sep 09 20:02:31.803466 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.803448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:31.806723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.806694    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 20:02:31.806832 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.806802    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 20:02:31.807594 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.807580    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-9qcdd\""
Sep 09 20:02:31.818791 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.818772    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sttnt"]
Sep 09 20:02:31.929091 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.929058    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtc2\" (UniqueName: \"kubernetes.io/projected/3b66747b-4bd6-4db6-9017-dd510d72080d-kube-api-access-fxtc2\") pod \"cert-manager-55b4d954d5-sttnt\" (UID: \"3b66747b-4bd6-4db6-9017-dd510d72080d\") " pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:31.929091 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:31.929098    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b66747b-4bd6-4db6-9017-dd510d72080d-bound-sa-token\") pod \"cert-manager-55b4d954d5-sttnt\" (UID: \"3b66747b-4bd6-4db6-9017-dd510d72080d\") " pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:32.029621 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.029590    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtc2\" (UniqueName: \"kubernetes.io/projected/3b66747b-4bd6-4db6-9017-dd510d72080d-kube-api-access-fxtc2\") pod \"cert-manager-55b4d954d5-sttnt\" (UID: \"3b66747b-4bd6-4db6-9017-dd510d72080d\") " pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:32.029787 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.029630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b66747b-4bd6-4db6-9017-dd510d72080d-bound-sa-token\") pod \"cert-manager-55b4d954d5-sttnt\" (UID: \"3b66747b-4bd6-4db6-9017-dd510d72080d\") " pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:32.039563 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.039531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtc2\" (UniqueName: \"kubernetes.io/projected/3b66747b-4bd6-4db6-9017-dd510d72080d-kube-api-access-fxtc2\") pod \"cert-manager-55b4d954d5-sttnt\" (UID: \"3b66747b-4bd6-4db6-9017-dd510d72080d\") " pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:32.039848 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.039831    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3b66747b-4bd6-4db6-9017-dd510d72080d-bound-sa-token\") pod \"cert-manager-55b4d954d5-sttnt\" (UID: \"3b66747b-4bd6-4db6-9017-dd510d72080d\") " pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:32.112087 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.112043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sttnt"
Sep 09 20:02:32.385588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.385486    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sttnt"]
Sep 09 20:02:32.387885 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:02:32.387850    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b66747b_4bd6_4db6_9017_dd510d72080d.slice/crio-69f67fd8847ae4c0c2b84575c892715961cd24f8402b70f7ac970092d1056cee WatchSource:0}: Error finding container 69f67fd8847ae4c0c2b84575c892715961cd24f8402b70f7ac970092d1056cee: Status 404 returned error can't find the container with id 69f67fd8847ae4c0c2b84575c892715961cd24f8402b70f7ac970092d1056cee
Sep 09 20:02:32.762047 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:32.761941    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sttnt" event={"ID":"3b66747b-4bd6-4db6-9017-dd510d72080d","Type":"ContainerStarted","Data":"69f67fd8847ae4c0c2b84575c892715961cd24f8402b70f7ac970092d1056cee"}
Sep 09 20:02:35.776939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:35.776905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sttnt" event={"ID":"3b66747b-4bd6-4db6-9017-dd510d72080d","Type":"ContainerStarted","Data":"915a969871fe1e03ec8fb2c11f3247b1274652680f4004f8e7cad9817c537c86"}
Sep 09 20:02:35.797094 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:35.797043    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-sttnt" podStartSLOduration=2.257948145 podStartE2EDuration="4.797027024s" totalImagesPullingTime="2.539078879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:35.795005746 +0000 UTC m=+458.313143273" watchObservedRunningTime="2026-09-09 20:02:35.797027024 +0000 UTC m=+458.315164551"
Sep 09 20:02:49.962791 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.962759    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"]
Sep 09 20:02:49.968235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.968216    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:49.972174 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.972154    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-psw7x\""
Sep 09 20:02:49.972737 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.972711    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:02:49.972837 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.972733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:02:49.972837 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.972799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:02:49.972949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.972837    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:02:49.986245 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:49.986224    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"]
Sep 09 20:02:50.068038 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.067995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.068038 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.068041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.068269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.068109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b006b6-13e3-4dd1-bf40-0382054f54f3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.068269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.068150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.068269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.068177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv27f\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-kube-api-access-fv27f\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.068269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.068249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.068465 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.068280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.168907 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.168866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169103 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.168917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b006b6-13e3-4dd1-bf40-0382054f54f3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169103 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.168946    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169103 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.168999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv27f\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-kube-api-access-fv27f\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169103 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.169059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169295 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.169193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169295 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.169248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.169966 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.169914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.171554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.171528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.171675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.171526    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b006b6-13e3-4dd1-bf40-0382054f54f3-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.171675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.171592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.171779 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.171723    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.179651 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.179627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.181737 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.181714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv27f\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-kube-api-access-fv27f\") pod \"istiod-openshift-gateway-7b7995d5cd-x99q2\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.277486 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.277387    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:50.555809 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.555782    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"]
Sep 09 20:02:50.557491 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:02:50.557464    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b006b6_13e3_4dd1_bf40_0382054f54f3.slice/crio-7b59381e5e5682a99b25821141f3782f1cdad1332a65e7f4bf6d41924b7d98fc WatchSource:0}: Error finding container 7b59381e5e5682a99b25821141f3782f1cdad1332a65e7f4bf6d41924b7d98fc: Status 404 returned error can't find the container with id 7b59381e5e5682a99b25821141f3782f1cdad1332a65e7f4bf6d41924b7d98fc
Sep 09 20:02:50.840441 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:50.840408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" event={"ID":"90b006b6-13e3-4dd1-bf40-0382054f54f3","Type":"ContainerStarted","Data":"7b59381e5e5682a99b25821141f3782f1cdad1332a65e7f4bf6d41924b7d98fc"}
Sep 09 20:02:53.533097 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:53.533061    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 20:02:53.533486 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:53.533150    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 20:02:53.864938 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:53.864903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" event={"ID":"90b006b6-13e3-4dd1-bf40-0382054f54f3","Type":"ContainerStarted","Data":"db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41"}
Sep 09 20:02:53.865134 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:53.865037    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:02:53.893661 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:53.892384    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" podStartSLOduration=1.9189438559999998 podStartE2EDuration="4.892362875s" totalImagesPullingTime="2.973419019s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:02:53.890634623 +0000 UTC m=+476.408772153" watchObservedRunningTime="2026-09-09 20:02:53.892362875 +0000 UTC m=+476.410500403"
Sep 09 20:02:54.871458 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:54.871424    2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x99q2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:02:54.871807 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:54.871477    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:02:57.870625 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:02:57.870598    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:03:10.890375 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.890338    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"]
Sep 09 20:03:10.893130 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.893115    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:10.896040 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.896016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rlsmd\""
Sep 09 20:03:10.896157 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.896024    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 09 20:03:10.896157 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.896047    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 09 20:03:10.903426 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.903406    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"]
Sep 09 20:03:10.936853 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.936823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:10.937033 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.936858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:10.937033 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.936908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsg5d\" (UniqueName: \"kubernetes.io/projected/95d839e3-1793-4ad7-a0a8-452fe5d67624-kube-api-access-dsg5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:10.987209 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.987179    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"]
Sep 09 20:03:10.989860 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.989845    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:10.998920 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:10.998887    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"]
Sep 09 20:03:11.037862 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.037827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.038077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.037868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcls6\" (UniqueName: \"kubernetes.io/projected/e6937bb9-da4f-4e78-a275-0114bb24805e-kube-api-access-dcls6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.038077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.037892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.038077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.037945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsg5d\" (UniqueName: \"kubernetes.io/projected/95d839e3-1793-4ad7-a0a8-452fe5d67624-kube-api-access-dsg5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.038077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.038012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.038302 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.038195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.038302 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.038205    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.038411 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.038364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.046837 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.046813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsg5d\" (UniqueName: \"kubernetes.io/projected/95d839e3-1793-4ad7-a0a8-452fe5d67624-kube-api-access-dsg5d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.085164 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.085137    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"]
Sep 09 20:03:11.087470 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.087456    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.099768 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.099744    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"]
Sep 09 20:03:11.139032 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.139032 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139033    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.139247 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.139247 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcls6\" (UniqueName: \"kubernetes.io/projected/e6937bb9-da4f-4e78-a275-0114bb24805e-kube-api-access-dcls6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.139247 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.139247 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139226    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqh5v\" (UniqueName: \"kubernetes.io/projected/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-kube-api-access-dqh5v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.139457 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.139457 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.139436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.148342 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.148284    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcls6\" (UniqueName: \"kubernetes.io/projected/e6937bb9-da4f-4e78-a275-0114bb24805e-kube-api-access-dcls6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.196989 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.196949    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"]
Sep 09 20:03:11.199270 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.199256    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.201989 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.201968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:11.214302 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.214281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"]
Sep 09 20:03:11.240066 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240034    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.240236 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.240236 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.240345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqh5v\" (UniqueName: \"kubernetes.io/projected/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-kube-api-access-dqh5v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.240345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d697c\" (UniqueName: \"kubernetes.io/projected/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-kube-api-access-d697c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.240445 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240342    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.240494 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.240579 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.240561    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.252511 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.251648    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqh5v\" (UniqueName: \"kubernetes.io/projected/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-kube-api-access-dqh5v\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.298458 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.298428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:11.341064 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.341029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.341269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.341085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.341269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.341147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d697c\" (UniqueName: \"kubernetes.io/projected/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-kube-api-access-d697c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.341492 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.341467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.341836 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.341816    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.364739 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.364703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d697c\" (UniqueName: \"kubernetes.io/projected/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-kube-api-access-d697c\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.396565 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.396532    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:11.480097 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.480072    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"]
Sep 09 20:03:11.482356 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:03:11.482332    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d839e3_1793_4ad7_a0a8_452fe5d67624.slice/crio-f6e6f28bc7d77ca93963855ade3869cfdaaad35d1d2c531fd466652d476b0338 WatchSource:0}: Error finding container f6e6f28bc7d77ca93963855ade3869cfdaaad35d1d2c531fd466652d476b0338: Status 404 returned error can't find the container with id f6e6f28bc7d77ca93963855ade3869cfdaaad35d1d2c531fd466652d476b0338
Sep 09 20:03:11.508321 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.508302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:11.571236 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.571209    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"]
Sep 09 20:03:11.574389 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:03:11.574356    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6937bb9_da4f_4e78_a275_0114bb24805e.slice/crio-50909c393e6ff0ce17691f1530cd2f3772f18dfd8755a6bfe1bd1ccf571c30ee WatchSource:0}: Error finding container 50909c393e6ff0ce17691f1530cd2f3772f18dfd8755a6bfe1bd1ccf571c30ee: Status 404 returned error can't find the container with id 50909c393e6ff0ce17691f1530cd2f3772f18dfd8755a6bfe1bd1ccf571c30ee
Sep 09 20:03:11.670397 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.670374    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"]
Sep 09 20:03:11.672510 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:03:11.672484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0615a2_39b0_4665_a6a9_3cd71788c3c5.slice/crio-7b32cab2fb17e6ac3fa32213007f7a6aa13acdaa87ce06af782fbe8af7c8f93b WatchSource:0}: Error finding container 7b32cab2fb17e6ac3fa32213007f7a6aa13acdaa87ce06af782fbe8af7c8f93b: Status 404 returned error can't find the container with id 7b32cab2fb17e6ac3fa32213007f7a6aa13acdaa87ce06af782fbe8af7c8f93b
Sep 09 20:03:11.782738 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.782717    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"]
Sep 09 20:03:11.784222 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:03:11.784203    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9d78e8_e8bc_4a85_9c36_62a34f3244d9.slice/crio-914360704e320687f4d35ab34caea9f81e65765749305b00795a87b470f9869d WatchSource:0}: Error finding container 914360704e320687f4d35ab34caea9f81e65765749305b00795a87b470f9869d: Status 404 returned error can't find the container with id 914360704e320687f4d35ab34caea9f81e65765749305b00795a87b470f9869d
Sep 09 20:03:11.946376 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.946339    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerID="b6e7c286d7ed202e26f50e5528cc818def9986d8c5a83a42add707e0fee1f287" exitCode=0
Sep 09 20:03:11.946748 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.946413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b" event={"ID":"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9","Type":"ContainerDied","Data":"b6e7c286d7ed202e26f50e5528cc818def9986d8c5a83a42add707e0fee1f287"}
Sep 09 20:03:11.946748 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.946458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b" event={"ID":"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9","Type":"ContainerStarted","Data":"914360704e320687f4d35ab34caea9f81e65765749305b00795a87b470f9869d"}
Sep 09 20:03:11.948201 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.948170    2576 generic.go:356] "Generic (PLEG): container finished" podID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerID="07d69221f77254e59198a99bb6e41ea378cb121d9e47f96937aad6dafac0ed44" exitCode=0
Sep 09 20:03:11.948298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.948246    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw" event={"ID":"95d839e3-1793-4ad7-a0a8-452fe5d67624","Type":"ContainerDied","Data":"07d69221f77254e59198a99bb6e41ea378cb121d9e47f96937aad6dafac0ed44"}
Sep 09 20:03:11.948298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.948281    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw" event={"ID":"95d839e3-1793-4ad7-a0a8-452fe5d67624","Type":"ContainerStarted","Data":"f6e6f28bc7d77ca93963855ade3869cfdaaad35d1d2c531fd466652d476b0338"}
Sep 09 20:03:11.949788 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.949764    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerID="9b5c33d4bcb2ed64d91cdcc6bd428424d9cb7ce93dfe5ad544732bd79d153ac2" exitCode=0
Sep 09 20:03:11.950054 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.949869    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg" event={"ID":"8b0615a2-39b0-4665-a6a9-3cd71788c3c5","Type":"ContainerDied","Data":"9b5c33d4bcb2ed64d91cdcc6bd428424d9cb7ce93dfe5ad544732bd79d153ac2"}
Sep 09 20:03:11.950054 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.949906    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg" event={"ID":"8b0615a2-39b0-4665-a6a9-3cd71788c3c5","Type":"ContainerStarted","Data":"7b32cab2fb17e6ac3fa32213007f7a6aa13acdaa87ce06af782fbe8af7c8f93b"}
Sep 09 20:03:11.951562 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.951543    2576 generic.go:356] "Generic (PLEG): container finished" podID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerID="894a58f6803b20e8f5f105feca5da9a096c0e06f91e3ed46b6f3a68c898fa43c" exitCode=0
Sep 09 20:03:11.951623 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.951581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn" event={"ID":"e6937bb9-da4f-4e78-a275-0114bb24805e","Type":"ContainerDied","Data":"894a58f6803b20e8f5f105feca5da9a096c0e06f91e3ed46b6f3a68c898fa43c"}
Sep 09 20:03:11.951623 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:11.951611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn" event={"ID":"e6937bb9-da4f-4e78-a275-0114bb24805e","Type":"ContainerStarted","Data":"50909c393e6ff0ce17691f1530cd2f3772f18dfd8755a6bfe1bd1ccf571c30ee"}
Sep 09 20:03:13.963385 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:13.963350    2576 generic.go:356] "Generic (PLEG): container finished" podID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerID="3bb2adb9ae0930de78b0b762ad06744561252a6bf348055f0686e9787b91d88d" exitCode=0
Sep 09 20:03:13.963852 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:13.963435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn" event={"ID":"e6937bb9-da4f-4e78-a275-0114bb24805e","Type":"ContainerDied","Data":"3bb2adb9ae0930de78b0b762ad06744561252a6bf348055f0686e9787b91d88d"}
Sep 09 20:03:13.965204 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:13.965183    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerID="289f2718e9596165161817f15be5d475042293d823ed2a7011e84c889cd952f1" exitCode=0
Sep 09 20:03:13.965309 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:13.965287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b" event={"ID":"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9","Type":"ContainerDied","Data":"289f2718e9596165161817f15be5d475042293d823ed2a7011e84c889cd952f1"}
Sep 09 20:03:13.967215 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:13.967150    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerID="75f1aabae7686e20831911d81af9b2005724481f93bfdcde9741b01b829204fb" exitCode=0
Sep 09 20:03:13.967215 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:13.967184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg" event={"ID":"8b0615a2-39b0-4665-a6a9-3cd71788c3c5","Type":"ContainerDied","Data":"75f1aabae7686e20831911d81af9b2005724481f93bfdcde9741b01b829204fb"}
Sep 09 20:03:14.974176 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:14.974142    2576 generic.go:356] "Generic (PLEG): container finished" podID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerID="6ac93e40f2ff259250ad1b8d23c21ec4fe132aafff766959bf51ec71fee39aef" exitCode=0
Sep 09 20:03:14.974613 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:14.974231    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b" event={"ID":"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9","Type":"ContainerDied","Data":"6ac93e40f2ff259250ad1b8d23c21ec4fe132aafff766959bf51ec71fee39aef"}
Sep 09 20:03:14.976255 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:14.976233    2576 generic.go:356] "Generic (PLEG): container finished" podID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerID="bd2e378ad497798f1cba63aee22a94323635f5035f6093903a54ea96fd7ca155" exitCode=0
Sep 09 20:03:14.976377 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:14.976308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg" event={"ID":"8b0615a2-39b0-4665-a6a9-3cd71788c3c5","Type":"ContainerDied","Data":"bd2e378ad497798f1cba63aee22a94323635f5035f6093903a54ea96fd7ca155"}
Sep 09 20:03:14.978410 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:14.978389    2576 generic.go:356] "Generic (PLEG): container finished" podID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerID="1fba862e94cb57ce34e10dcc5da8067b38122c6b0e0300c21e61dfa66353a6e0" exitCode=0
Sep 09 20:03:14.978517 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:14.978417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn" event={"ID":"e6937bb9-da4f-4e78-a275-0114bb24805e","Type":"ContainerDied","Data":"1fba862e94cb57ce34e10dcc5da8067b38122c6b0e0300c21e61dfa66353a6e0"}
Sep 09 20:03:16.127112 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.127088    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:16.167424 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.167401    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:16.170269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.170248    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:16.181590 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.181570    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d697c\" (UniqueName: \"kubernetes.io/projected/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-kube-api-access-d697c\") pod \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") "
Sep 09 20:03:16.181695 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.181632    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-util\") pod \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") "
Sep 09 20:03:16.181751 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.181719    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-bundle\") pod \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\" (UID: \"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9\") "
Sep 09 20:03:16.182190 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.182167    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-bundle" (OuterVolumeSpecName: "bundle") pod "ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" (UID: "ec9d78e8-e8bc-4a85-9c36-62a34f3244d9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:16.183741 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.183720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-kube-api-access-d697c" (OuterVolumeSpecName: "kube-api-access-d697c") pod "ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" (UID: "ec9d78e8-e8bc-4a85-9c36-62a34f3244d9"). InnerVolumeSpecName "kube-api-access-d697c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:03:16.186643 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.186623    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-util" (OuterVolumeSpecName: "util") pod "ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" (UID: "ec9d78e8-e8bc-4a85-9c36-62a34f3244d9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:16.282527 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcls6\" (UniqueName: \"kubernetes.io/projected/e6937bb9-da4f-4e78-a275-0114bb24805e-kube-api-access-dcls6\") pod \"e6937bb9-da4f-4e78-a275-0114bb24805e\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") "
Sep 09 20:03:16.282527 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282493    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-util\") pod \"e6937bb9-da4f-4e78-a275-0114bb24805e\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") "
Sep 09 20:03:16.282527 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282512    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqh5v\" (UniqueName: \"kubernetes.io/projected/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-kube-api-access-dqh5v\") pod \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") "
Sep 09 20:03:16.282768 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-util\") pod \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") "
Sep 09 20:03:16.282768 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282608    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-bundle\") pod \"e6937bb9-da4f-4e78-a275-0114bb24805e\" (UID: \"e6937bb9-da4f-4e78-a275-0114bb24805e\") "
Sep 09 20:03:16.282768 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-bundle\") pod \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\" (UID: \"8b0615a2-39b0-4665-a6a9-3cd71788c3c5\") "
Sep 09 20:03:16.282909 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282767    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d697c\" (UniqueName: \"kubernetes.io/projected/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-kube-api-access-d697c\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.282909 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282784    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.282909 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.282797    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ec9d78e8-e8bc-4a85-9c36-62a34f3244d9-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.283413 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.283243    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-bundle" (OuterVolumeSpecName: "bundle") pod "e6937bb9-da4f-4e78-a275-0114bb24805e" (UID: "e6937bb9-da4f-4e78-a275-0114bb24805e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:16.283551 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.283526    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-bundle" (OuterVolumeSpecName: "bundle") pod "8b0615a2-39b0-4665-a6a9-3cd71788c3c5" (UID: "8b0615a2-39b0-4665-a6a9-3cd71788c3c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:16.284817 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.284795    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6937bb9-da4f-4e78-a275-0114bb24805e-kube-api-access-dcls6" (OuterVolumeSpecName: "kube-api-access-dcls6") pod "e6937bb9-da4f-4e78-a275-0114bb24805e" (UID: "e6937bb9-da4f-4e78-a275-0114bb24805e"). InnerVolumeSpecName "kube-api-access-dcls6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:03:16.285068 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.285051    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-kube-api-access-dqh5v" (OuterVolumeSpecName: "kube-api-access-dqh5v") pod "8b0615a2-39b0-4665-a6a9-3cd71788c3c5" (UID: "8b0615a2-39b0-4665-a6a9-3cd71788c3c5"). InnerVolumeSpecName "kube-api-access-dqh5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:03:16.288453 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.288414    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-util" (OuterVolumeSpecName: "util") pod "e6937bb9-da4f-4e78-a275-0114bb24805e" (UID: "e6937bb9-da4f-4e78-a275-0114bb24805e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:16.288519 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.288494    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-util" (OuterVolumeSpecName: "util") pod "8b0615a2-39b0-4665-a6a9-3cd71788c3c5" (UID: "8b0615a2-39b0-4665-a6a9-3cd71788c3c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:16.383444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.383409    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.383444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.383437    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.383444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.383446    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcls6\" (UniqueName: \"kubernetes.io/projected/e6937bb9-da4f-4e78-a275-0114bb24805e-kube-api-access-dcls6\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.383666 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.383456    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e6937bb9-da4f-4e78-a275-0114bb24805e-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.383666 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.383465    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqh5v\" (UniqueName: \"kubernetes.io/projected/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-kube-api-access-dqh5v\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.383666 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.383473    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8b0615a2-39b0-4665-a6a9-3cd71788c3c5-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:16.987877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.987845    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b"
Sep 09 20:03:16.988052 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.987874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76t97b" event={"ID":"ec9d78e8-e8bc-4a85-9c36-62a34f3244d9","Type":"ContainerDied","Data":"914360704e320687f4d35ab34caea9f81e65765749305b00795a87b470f9869d"}
Sep 09 20:03:16.988052 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.987925    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="914360704e320687f4d35ab34caea9f81e65765749305b00795a87b470f9869d"
Sep 09 20:03:16.989677 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.989658    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg"
Sep 09 20:03:16.989792 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.989673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847acszmg" event={"ID":"8b0615a2-39b0-4665-a6a9-3cd71788c3c5","Type":"ContainerDied","Data":"7b32cab2fb17e6ac3fa32213007f7a6aa13acdaa87ce06af782fbe8af7c8f93b"}
Sep 09 20:03:16.989792 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.989707    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b32cab2fb17e6ac3fa32213007f7a6aa13acdaa87ce06af782fbe8af7c8f93b"
Sep 09 20:03:16.995043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.995010    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn" event={"ID":"e6937bb9-da4f-4e78-a275-0114bb24805e","Type":"ContainerDied","Data":"50909c393e6ff0ce17691f1530cd2f3772f18dfd8755a6bfe1bd1ccf571c30ee"}
Sep 09 20:03:16.995167 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.995049    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50909c393e6ff0ce17691f1530cd2f3772f18dfd8755a6bfe1bd1ccf571c30ee"
Sep 09 20:03:16.995167 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:16.995052    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6rkvhn"
Sep 09 20:03:20.010089 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:20.010057    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw" event={"ID":"95d839e3-1793-4ad7-a0a8-452fe5d67624","Type":"ContainerStarted","Data":"5efda483a4db54c40261c0ae715cc3f467923f268b78932206a5cc407928a062"}
Sep 09 20:03:21.016701 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:21.016669    2576 generic.go:356] "Generic (PLEG): container finished" podID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerID="5efda483a4db54c40261c0ae715cc3f467923f268b78932206a5cc407928a062" exitCode=0
Sep 09 20:03:21.017184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:21.016750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw" event={"ID":"95d839e3-1793-4ad7-a0a8-452fe5d67624","Type":"ContainerDied","Data":"5efda483a4db54c40261c0ae715cc3f467923f268b78932206a5cc407928a062"}
Sep 09 20:03:22.022921 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:22.022884    2576 generic.go:356] "Generic (PLEG): container finished" podID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerID="b061f486266c796b27a4cdfe4fd296bb02c6a107e1326101cc10478942428475" exitCode=0
Sep 09 20:03:22.023368 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:22.022971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw" event={"ID":"95d839e3-1793-4ad7-a0a8-452fe5d67624","Type":"ContainerDied","Data":"b061f486266c796b27a4cdfe4fd296bb02c6a107e1326101cc10478942428475"}
Sep 09 20:03:23.158743 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.158719    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:23.234484 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.234444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dsg5d\" (UniqueName: \"kubernetes.io/projected/95d839e3-1793-4ad7-a0a8-452fe5d67624-kube-api-access-dsg5d\") pod \"95d839e3-1793-4ad7-a0a8-452fe5d67624\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") "
Sep 09 20:03:23.234484 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.234489    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-bundle\") pod \"95d839e3-1793-4ad7-a0a8-452fe5d67624\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") "
Sep 09 20:03:23.234748 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.234564    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-util\") pod \"95d839e3-1793-4ad7-a0a8-452fe5d67624\" (UID: \"95d839e3-1793-4ad7-a0a8-452fe5d67624\") "
Sep 09 20:03:23.235119 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.235097    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-bundle" (OuterVolumeSpecName: "bundle") pod "95d839e3-1793-4ad7-a0a8-452fe5d67624" (UID: "95d839e3-1793-4ad7-a0a8-452fe5d67624"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:23.236730 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.236706    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d839e3-1793-4ad7-a0a8-452fe5d67624-kube-api-access-dsg5d" (OuterVolumeSpecName: "kube-api-access-dsg5d") pod "95d839e3-1793-4ad7-a0a8-452fe5d67624" (UID: "95d839e3-1793-4ad7-a0a8-452fe5d67624"). InnerVolumeSpecName "kube-api-access-dsg5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:03:23.239020 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.238982    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-util" (OuterVolumeSpecName: "util") pod "95d839e3-1793-4ad7-a0a8-452fe5d67624" (UID: "95d839e3-1793-4ad7-a0a8-452fe5d67624"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:03:23.336179 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.336083    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsg5d\" (UniqueName: \"kubernetes.io/projected/95d839e3-1793-4ad7-a0a8-452fe5d67624-kube-api-access-dsg5d\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:23.336179 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.336125    2576 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-bundle\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:23.336179 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:23.336135    2576 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/95d839e3-1793-4ad7-a0a8-452fe5d67624-util\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:03:24.033814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:24.033778    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw" event={"ID":"95d839e3-1793-4ad7-a0a8-452fe5d67624","Type":"ContainerDied","Data":"f6e6f28bc7d77ca93963855ade3869cfdaaad35d1d2c531fd466652d476b0338"}
Sep 09 20:03:24.034032 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:24.033833    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6e6f28bc7d77ca93963855ade3869cfdaaad35d1d2c531fd466652d476b0338"
Sep 09 20:03:24.034032 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:24.033788    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cxq7sw"
Sep 09 20:03:33.532342 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532308    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"]
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532605    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532615    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532627    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="extract"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532633    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="extract"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532641    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532646    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532652    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532657    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532664    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532669    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532675    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="extract"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532680    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="extract"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532687    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532692    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532700    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532704    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="pull"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532714    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532718    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532723    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532728    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="util"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532734    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="extract"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532739    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="extract"
Sep 09 20:03:33.532733 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532744    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="extract"
Sep 09 20:03:33.533519 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532749    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="extract"
Sep 09 20:03:33.533519 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532815    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9d78e8-e8bc-4a85-9c36-62a34f3244d9" containerName="extract"
Sep 09 20:03:33.533519 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532823    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="95d839e3-1793-4ad7-a0a8-452fe5d67624" containerName="extract"
Sep 09 20:03:33.533519 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532829    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b0615a2-39b0-4665-a6a9-3cd71788c3c5" containerName="extract"
Sep 09 20:03:33.533519 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.532834    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6937bb9-da4f-4e78-a275-0114bb24805e" containerName="extract"
Sep 09 20:03:33.541413 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.541393    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.545738 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.545715    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:03:33.545856 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.545751    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:03:33.545856 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.545764    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-z5wp5\""
Sep 09 20:03:33.549833 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.549806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"]
Sep 09 20:03:33.618172 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.618138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36c80767-9be6-4fc8-90be-8ca5b1ad360f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-4gdsf\" (UID: \"36c80767-9be6-4fc8-90be-8ca5b1ad360f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.618329 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.618188    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qpw\" (UniqueName: \"kubernetes.io/projected/36c80767-9be6-4fc8-90be-8ca5b1ad360f-kube-api-access-n2qpw\") pod \"kuadrant-operator-controller-manager-6d4cb79c-4gdsf\" (UID: \"36c80767-9be6-4fc8-90be-8ca5b1ad360f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.719449 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.719411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qpw\" (UniqueName: \"kubernetes.io/projected/36c80767-9be6-4fc8-90be-8ca5b1ad360f-kube-api-access-n2qpw\") pod \"kuadrant-operator-controller-manager-6d4cb79c-4gdsf\" (UID: \"36c80767-9be6-4fc8-90be-8ca5b1ad360f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.719614 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.719472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36c80767-9be6-4fc8-90be-8ca5b1ad360f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-4gdsf\" (UID: \"36c80767-9be6-4fc8-90be-8ca5b1ad360f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.719815 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.719798    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/36c80767-9be6-4fc8-90be-8ca5b1ad360f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-4gdsf\" (UID: \"36c80767-9be6-4fc8-90be-8ca5b1ad360f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.751236 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.751207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qpw\" (UniqueName: \"kubernetes.io/projected/36c80767-9be6-4fc8-90be-8ca5b1ad360f-kube-api-access-n2qpw\") pod \"kuadrant-operator-controller-manager-6d4cb79c-4gdsf\" (UID: \"36c80767-9be6-4fc8-90be-8ca5b1ad360f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:33.852080 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:33.852048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:34.131796 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:34.131576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"]
Sep 09 20:03:34.133790 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:03:34.133765    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c80767_9be6_4fc8_90be_8ca5b1ad360f.slice/crio-cfc77df0db092736518513bfa2cf1b7266267cb995825a5b4fba0b4017df04cf WatchSource:0}: Error finding container cfc77df0db092736518513bfa2cf1b7266267cb995825a5b4fba0b4017df04cf: Status 404 returned error can't find the container with id cfc77df0db092736518513bfa2cf1b7266267cb995825a5b4fba0b4017df04cf
Sep 09 20:03:35.085020 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:35.084985    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf" event={"ID":"36c80767-9be6-4fc8-90be-8ca5b1ad360f","Type":"ContainerStarted","Data":"cfc77df0db092736518513bfa2cf1b7266267cb995825a5b4fba0b4017df04cf"}
Sep 09 20:03:36.086050 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.086011    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"]
Sep 09 20:03:36.089499 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.089469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:03:36.092483 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.092464    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-tc77n\""
Sep 09 20:03:36.104317 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.104291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"]
Sep 09 20:03:36.242930 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.242892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8hbk\" (UniqueName: \"kubernetes.io/projected/c3390f21-57b3-4b82-9cf7-1a8ba4754abd-kube-api-access-s8hbk\") pod \"authorino-operator-6fc8f864f4-2p6c6\" (UID: \"c3390f21-57b3-4b82-9cf7-1a8ba4754abd\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:03:36.343808 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.343766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8hbk\" (UniqueName: \"kubernetes.io/projected/c3390f21-57b3-4b82-9cf7-1a8ba4754abd-kube-api-access-s8hbk\") pod \"authorino-operator-6fc8f864f4-2p6c6\" (UID: \"c3390f21-57b3-4b82-9cf7-1a8ba4754abd\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:03:36.354169 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.354137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8hbk\" (UniqueName: \"kubernetes.io/projected/c3390f21-57b3-4b82-9cf7-1a8ba4754abd-kube-api-access-s8hbk\") pod \"authorino-operator-6fc8f864f4-2p6c6\" (UID: \"c3390f21-57b3-4b82-9cf7-1a8ba4754abd\") " pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:03:36.401943 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.401906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:03:36.692084 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:36.691977    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"]
Sep 09 20:03:36.696129 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:03:36.696096    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3390f21_57b3_4b82_9cf7_1a8ba4754abd.slice/crio-3b58560b4e18582e681835edf8805efd99578c019f855fed2d77ee94f5ff2176 WatchSource:0}: Error finding container 3b58560b4e18582e681835edf8805efd99578c019f855fed2d77ee94f5ff2176: Status 404 returned error can't find the container with id 3b58560b4e18582e681835edf8805efd99578c019f855fed2d77ee94f5ff2176
Sep 09 20:03:37.098239 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:37.098202    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6" event={"ID":"c3390f21-57b3-4b82-9cf7-1a8ba4754abd","Type":"ContainerStarted","Data":"3b58560b4e18582e681835edf8805efd99578c019f855fed2d77ee94f5ff2176"}
Sep 09 20:03:39.111032 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:39.110995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf" event={"ID":"36c80767-9be6-4fc8-90be-8ca5b1ad360f","Type":"ContainerStarted","Data":"29d473072ec706eaaf3f57fa97b16232f9aa8c3871d2943e8d66f71e1822e176"}
Sep 09 20:03:39.111407 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:39.111173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:39.174137 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:39.174084    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf" podStartSLOduration=1.6599498719999999 podStartE2EDuration="6.174069549s" totalImagesPullingTime="4.514119677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:39.173880534 +0000 UTC m=+521.692018061" watchObservedRunningTime="2026-09-09 20:03:39.174069549 +0000 UTC m=+521.692207073"
Sep 09 20:03:40.118299 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:40.118257    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6" event={"ID":"c3390f21-57b3-4b82-9cf7-1a8ba4754abd","Type":"ContainerStarted","Data":"d6dd8db8611a60491f870b9232aafcb46f042bd8770cd1b6422cdf684bd1ef29"}
Sep 09 20:03:40.144833 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:40.144781    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6" podStartSLOduration=0.905405503 podStartE2EDuration="4.144766072s" totalImagesPullingTime="3.239360569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:03:40.1427156 +0000 UTC m=+522.660853131" watchObservedRunningTime="2026-09-09 20:03:40.144766072 +0000 UTC m=+522.662903598"
Sep 09 20:03:41.125047 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:41.125013    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:03:50.120867 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:50.120786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-4gdsf"
Sep 09 20:03:52.131908 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:03:52.131880    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6fc8f864f4-2p6c6"
Sep 09 20:04:23.766828 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.766794    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:23.769104 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.769076    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:23.771742 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.771724    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zkj67\""
Sep 09 20:04:23.771859 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.771765    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 20:04:23.780680 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.780652    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:23.859607 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.859572    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:23.930828 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.930794    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9fe6a13e-e484-457e-895e-db16a3712084-config-file\") pod \"limitador-limitador-8588749977-gjl4l\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") " pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:23.931029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:23.930838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgnf\" (UniqueName: \"kubernetes.io/projected/9fe6a13e-e484-457e-895e-db16a3712084-kube-api-access-zzgnf\") pod \"limitador-limitador-8588749977-gjl4l\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") " pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:24.031683 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:24.031586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgnf\" (UniqueName: \"kubernetes.io/projected/9fe6a13e-e484-457e-895e-db16a3712084-kube-api-access-zzgnf\") pod \"limitador-limitador-8588749977-gjl4l\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") " pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:24.031854 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:24.031682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9fe6a13e-e484-457e-895e-db16a3712084-config-file\") pod \"limitador-limitador-8588749977-gjl4l\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") " pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:24.032359 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:24.032334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9fe6a13e-e484-457e-895e-db16a3712084-config-file\") pod \"limitador-limitador-8588749977-gjl4l\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") " pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:24.044187 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:24.044160    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgnf\" (UniqueName: \"kubernetes.io/projected/9fe6a13e-e484-457e-895e-db16a3712084-kube-api-access-zzgnf\") pod \"limitador-limitador-8588749977-gjl4l\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") " pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:24.079716 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:24.079687    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:24.353251 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:24.353228    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:24.355503 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:04:24.355473    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe6a13e_e484_457e_895e_db16a3712084.slice/crio-9adf21c833ce450a25059cdaa4fa872e16169e4ccf3aa80cbd6301e56e5948fe WatchSource:0}: Error finding container 9adf21c833ce450a25059cdaa4fa872e16169e4ccf3aa80cbd6301e56e5948fe: Status 404 returned error can't find the container with id 9adf21c833ce450a25059cdaa4fa872e16169e4ccf3aa80cbd6301e56e5948fe
Sep 09 20:04:25.334260 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:25.334221    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l" event={"ID":"9fe6a13e-e484-457e-895e-db16a3712084","Type":"ContainerStarted","Data":"9adf21c833ce450a25059cdaa4fa872e16169e4ccf3aa80cbd6301e56e5948fe"}
Sep 09 20:04:26.340589 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:26.340553    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l" event={"ID":"9fe6a13e-e484-457e-895e-db16a3712084","Type":"ContainerStarted","Data":"668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36"}
Sep 09 20:04:26.340982 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:26.340628    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:26.361489 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:26.361427    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l" podStartSLOduration=2.239142412 podStartE2EDuration="3.361412393s" totalImagesPullingTime="1.122269981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:04:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:04:26.359511896 +0000 UTC m=+568.877649423" watchObservedRunningTime="2026-09-09 20:04:26.361412393 +0000 UTC m=+568.879549919"
Sep 09 20:04:37.346974 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:37.346917    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:37.723880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:37.723795    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:37.724046 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:37.724015    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l" podUID="9fe6a13e-e484-457e-895e-db16a3712084" containerName="limitador" containerID="cri-o://668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36" gracePeriod=30
Sep 09 20:04:38.272076 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.272050    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:38.347376 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.347351    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9fe6a13e-e484-457e-895e-db16a3712084-config-file\") pod \"9fe6a13e-e484-457e-895e-db16a3712084\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") "
Sep 09 20:04:38.347802 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.347391    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzgnf\" (UniqueName: \"kubernetes.io/projected/9fe6a13e-e484-457e-895e-db16a3712084-kube-api-access-zzgnf\") pod \"9fe6a13e-e484-457e-895e-db16a3712084\" (UID: \"9fe6a13e-e484-457e-895e-db16a3712084\") "
Sep 09 20:04:38.347802 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.347677    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fe6a13e-e484-457e-895e-db16a3712084-config-file" (OuterVolumeSpecName: "config-file") pod "9fe6a13e-e484-457e-895e-db16a3712084" (UID: "9fe6a13e-e484-457e-895e-db16a3712084"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:04:38.349645 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.349629    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fe6a13e-e484-457e-895e-db16a3712084-kube-api-access-zzgnf" (OuterVolumeSpecName: "kube-api-access-zzgnf") pod "9fe6a13e-e484-457e-895e-db16a3712084" (UID: "9fe6a13e-e484-457e-895e-db16a3712084"). InnerVolumeSpecName "kube-api-access-zzgnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:04:38.398297 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.398267    2576 generic.go:356] "Generic (PLEG): container finished" podID="9fe6a13e-e484-457e-895e-db16a3712084" containerID="668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36" exitCode=0
Sep 09 20:04:38.398422 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.398313    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l"
Sep 09 20:04:38.398422 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.398337    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l" event={"ID":"9fe6a13e-e484-457e-895e-db16a3712084","Type":"ContainerDied","Data":"668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36"}
Sep 09 20:04:38.398422 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.398376    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-gjl4l" event={"ID":"9fe6a13e-e484-457e-895e-db16a3712084","Type":"ContainerDied","Data":"9adf21c833ce450a25059cdaa4fa872e16169e4ccf3aa80cbd6301e56e5948fe"}
Sep 09 20:04:38.398422 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.398396    2576 scope.go:117] "RemoveContainer" containerID="668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36"
Sep 09 20:04:38.409736 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.409717    2576 scope.go:117] "RemoveContainer" containerID="668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36"
Sep 09 20:04:38.410015 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:04:38.409993    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36\": container with ID starting with 668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36 not found: ID does not exist" containerID="668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36"
Sep 09 20:04:38.410066 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.410023    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36"} err="failed to get container status \"668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36\": rpc error: code = NotFound desc = could not find container \"668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36\": container with ID starting with 668ea629ffb76226d2d8aa087c70d8dfbae1fd35aad8122bd2fa42b0d5e99b36 not found: ID does not exist"
Sep 09 20:04:38.422109 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.422088    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:38.426004 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.425982    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-gjl4l"]
Sep 09 20:04:38.448279 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.448261    2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/9fe6a13e-e484-457e-895e-db16a3712084-config-file\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:04:38.448279 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:38.448280    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgnf\" (UniqueName: \"kubernetes.io/projected/9fe6a13e-e484-457e-895e-db16a3712084-kube-api-access-zzgnf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:04:39.962825 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:39.962789    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fe6a13e-e484-457e-895e-db16a3712084" path="/var/lib/kubelet/pods/9fe6a13e-e484-457e-895e-db16a3712084/volumes"
Sep 09 20:04:57.896462 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:57.896433    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:04:57.899406 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:04:57.899382    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:05:07.942238 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:07.942198    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"]
Sep 09 20:05:07.942710 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:07.942599    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fe6a13e-e484-457e-895e-db16a3712084" containerName="limitador"
Sep 09 20:05:07.942710 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:07.942616    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fe6a13e-e484-457e-895e-db16a3712084" containerName="limitador"
Sep 09 20:05:07.942710 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:07.942709    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fe6a13e-e484-457e-895e-db16a3712084" containerName="limitador"
Sep 09 20:05:07.947645 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:07.947623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:07.962617 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:07.962592    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"]
Sep 09 20:05:08.090572 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.090572 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.090806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmcz\" (UniqueName: \"kubernetes.io/projected/5ccd73c4-40c5-401e-8d91-7171e03ae27a-kube-api-access-jgmcz\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.090806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.090806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccd73c4-40c5-401e-8d91-7171e03ae27a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.090806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090741    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.090806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.090793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191417 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmcz\" (UniqueName: \"kubernetes.io/projected/5ccd73c4-40c5-401e-8d91-7171e03ae27a-kube-api-access-jgmcz\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.191588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.191538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccd73c4-40c5-401e-8d91-7171e03ae27a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.192339 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.192275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.193968 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.193932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ccd73c4-40c5-401e-8d91-7171e03ae27a-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.194076 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.194060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.194343 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.194324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.194449 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.194429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.205476 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.205443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmcz\" (UniqueName: \"kubernetes.io/projected/5ccd73c4-40c5-401e-8d91-7171e03ae27a-kube-api-access-jgmcz\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.206603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.206585    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ccd73c4-40c5-401e-8d91-7171e03ae27a-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pcrbh\" (UID: \"5ccd73c4-40c5-401e-8d91-7171e03ae27a\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.255894 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.255866    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:08.535142 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.535109    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"]
Sep 09 20:05:08.536156 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:05:08.536134    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ccd73c4_40c5_401e_8d91_7171e03ae27a.slice/crio-933a35a71eb00881b503df9c45b2f4b57ab3e721493b64ffdc2f1c5b3d23a05b WatchSource:0}: Error finding container 933a35a71eb00881b503df9c45b2f4b57ab3e721493b64ffdc2f1c5b3d23a05b: Status 404 returned error can't find the container with id 933a35a71eb00881b503df9c45b2f4b57ab3e721493b64ffdc2f1c5b3d23a05b
Sep 09 20:05:08.538260 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.538228    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 20:05:08.538326 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.538299    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"}
Sep 09 20:05:08.545791 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:08.545768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh" event={"ID":"5ccd73c4-40c5-401e-8d91-7171e03ae27a","Type":"ContainerStarted","Data":"933a35a71eb00881b503df9c45b2f4b57ab3e721493b64ffdc2f1c5b3d23a05b"}
Sep 09 20:05:09.551575 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.551541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh" event={"ID":"5ccd73c4-40c5-401e-8d91-7171e03ae27a","Type":"ContainerStarted","Data":"9e7e4b4fb0cca194c4d740244e8df26270fd99c4f0e48cd39fd5c11c684f53c0"}
Sep 09 20:05:09.552043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.551723    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:09.553544 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.553522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh"
Sep 09 20:05:09.574693 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.574652    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pcrbh" podStartSLOduration=2.5746389880000002 podStartE2EDuration="2.574638988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:09.573023903 +0000 UTC m=+612.091161432" watchObservedRunningTime="2026-09-09 20:05:09.574638988 +0000 UTC m=+612.092776514"
Sep 09 20:05:09.635829 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.635801    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"]
Sep 09 20:05:09.636088 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.636064    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerName="discovery" containerID="cri-o://db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41" gracePeriod=30
Sep 09 20:05:09.889409 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:09.889383    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:05:10.007696 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007660    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-cacerts\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.007877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007720    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-kubeconfig\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.007877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007765    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-token\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.007877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007795    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b006b6-13e3-4dd1-bf40-0382054f54f3-local-certs\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.007877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007823    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-ca-configmap\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.007877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007858    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv27f\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-kube-api-access-fv27f\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.008186 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.007886    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-dns-cert\") pod \"90b006b6-13e3-4dd1-bf40-0382054f54f3\" (UID: \"90b006b6-13e3-4dd1-bf40-0382054f54f3\") "
Sep 09 20:05:10.008297 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.008267    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:05:10.010318 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.010283    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-kube-api-access-fv27f" (OuterVolumeSpecName: "kube-api-access-fv27f") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "kube-api-access-fv27f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:05:10.010434 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.010322    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-cacerts" (OuterVolumeSpecName: "cacerts") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:05:10.010493 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.010457    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:05:10.010654 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.010620    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:05:10.010654 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.010630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:05:10.010788 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.010676    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90b006b6-13e3-4dd1-bf40-0382054f54f3-local-certs" (OuterVolumeSpecName: "local-certs") pod "90b006b6-13e3-4dd1-bf40-0382054f54f3" (UID: "90b006b6-13e3-4dd1-bf40-0382054f54f3"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:05:10.109250 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109216    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-token\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.109250 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109252    2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/90b006b6-13e3-4dd1-bf40-0382054f54f3-local-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.109250 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109267    2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-ca-configmap\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.109545 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109285    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv27f\" (UniqueName: \"kubernetes.io/projected/90b006b6-13e3-4dd1-bf40-0382054f54f3-kube-api-access-fv27f\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.109545 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109301    2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-csr-dns-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.109545 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109315    2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-cacerts\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.109545 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.109330    2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/90b006b6-13e3-4dd1-bf40-0382054f54f3-istio-kubeconfig\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:05:10.557980 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.557866    2576 generic.go:356] "Generic (PLEG): container finished" podID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerID="db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41" exitCode=0
Sep 09 20:05:10.557980 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.557915    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"
Sep 09 20:05:10.557980 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.557971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" event={"ID":"90b006b6-13e3-4dd1-bf40-0382054f54f3","Type":"ContainerDied","Data":"db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41"}
Sep 09 20:05:10.558502 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.558014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" event={"ID":"90b006b6-13e3-4dd1-bf40-0382054f54f3","Type":"ContainerDied","Data":"7b59381e5e5682a99b25821141f3782f1cdad1332a65e7f4bf6d41924b7d98fc"}
Sep 09 20:05:10.558502 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.558033    2576 scope.go:117] "RemoveContainer" containerID="db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41"
Sep 09 20:05:10.567275 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.567261    2576 scope.go:117] "RemoveContainer" containerID="db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41"
Sep 09 20:05:10.567530 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:05:10.567515    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41\": container with ID starting with db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41 not found: ID does not exist" containerID="db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41"
Sep 09 20:05:10.567582 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.567538    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41"} err="failed to get container status \"db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41\": rpc error: code = NotFound desc = could not find container \"db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41\": container with ID starting with db7232b1fbdaebf66dd12c7cfaf55936a302178d2cb506ba2d9e10536de61f41 not found: ID does not exist"
Sep 09 20:05:10.579325 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.579301    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"]
Sep 09 20:05:10.584736 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:10.584715    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2"]
Sep 09 20:05:11.961495 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:11.961454    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" path="/var/lib/kubelet/pods/90b006b6-13e3-4dd1-bf40-0382054f54f3/volumes"
Sep 09 20:05:14.871091 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:14.871042    2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-x99q2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 20:05:14.871488 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:14.871146    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-x99q2" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.25:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 20:05:22.330714 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.330643    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-64lcz"]
Sep 09 20:05:22.331093 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.330918    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerName="discovery"
Sep 09 20:05:22.331093 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.330928    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerName="discovery"
Sep 09 20:05:22.331093 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.331004    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b006b6-13e3-4dd1-bf40-0382054f54f3" containerName="discovery"
Sep 09 20:05:22.333641 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.333626    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:22.337548 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.337530    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 09 20:05:22.337818 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.337801    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 09 20:05:22.337873 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.337858    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-7fvd4\""
Sep 09 20:05:22.337918 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.337879    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 09 20:05:22.348970 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.348932    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-64lcz"]
Sep 09 20:05:22.362233 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.362206    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"]
Sep 09 20:05:22.365287 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.365269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:22.367807 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.367786    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:05:22.368040 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.368025    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-lfk8h\""
Sep 09 20:05:22.379381 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.379359    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"]
Sep 09 20:05:22.514411 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.514380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92a1c2ba-add6-4467-877a-f2cb39701927-cert\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:22.514603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.514417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk649\" (UniqueName: \"kubernetes.io/projected/0b780fd5-3f70-460d-a236-ce244b4359d5-kube-api-access-fk649\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:22.514603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.514444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5dr\" (UniqueName: \"kubernetes.io/projected/92a1c2ba-add6-4467-877a-f2cb39701927-kube-api-access-zk5dr\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:22.514603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.514526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:22.615093 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.615061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:22.615286 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.615146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92a1c2ba-add6-4467-877a-f2cb39701927-cert\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:22.615286 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.615172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk649\" (UniqueName: \"kubernetes.io/projected/0b780fd5-3f70-460d-a236-ce244b4359d5-kube-api-access-fk649\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:22.615286 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:05:22.615201    2576 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 09 20:05:22.615286 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:05:22.615285    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert podName:0b780fd5-3f70-460d-a236-ce244b4359d5 nodeName:}" failed. No retries permitted until 2026-09-09 20:05:23.115268627 +0000 UTC m=+625.633406133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert") pod "kserve-controller-manager-7676c798d9-64lcz" (UID: "0b780fd5-3f70-460d-a236-ce244b4359d5") : secret "kserve-webhook-server-cert" not found
Sep 09 20:05:22.615503 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:05:22.615291    2576 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 09 20:05:22.615503 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.615204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5dr\" (UniqueName: \"kubernetes.io/projected/92a1c2ba-add6-4467-877a-f2cb39701927-kube-api-access-zk5dr\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:22.615503 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:05:22.615349    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92a1c2ba-add6-4467-877a-f2cb39701927-cert podName:92a1c2ba-add6-4467-877a-f2cb39701927 nodeName:}" failed. No retries permitted until 2026-09-09 20:05:23.115334069 +0000 UTC m=+625.633471589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/92a1c2ba-add6-4467-877a-f2cb39701927-cert") pod "llmisvc-controller-manager-59449d9fbd-kl5hd" (UID: "92a1c2ba-add6-4467-877a-f2cb39701927") : secret "llmisvc-webhook-server-cert" not found
Sep 09 20:05:22.629467 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.629432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5dr\" (UniqueName: \"kubernetes.io/projected/92a1c2ba-add6-4467-877a-f2cb39701927-kube-api-access-zk5dr\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:22.630009 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:22.629993    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk649\" (UniqueName: \"kubernetes.io/projected/0b780fd5-3f70-460d-a236-ce244b4359d5-kube-api-access-fk649\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:23.119498 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.119459    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92a1c2ba-add6-4467-877a-f2cb39701927-cert\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:23.119690 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.119536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:23.121993 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.121971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert\") pod \"kserve-controller-manager-7676c798d9-64lcz\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") " pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:23.122096 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.122040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/92a1c2ba-add6-4467-877a-f2cb39701927-cert\") pod \"llmisvc-controller-manager-59449d9fbd-kl5hd\" (UID: \"92a1c2ba-add6-4467-877a-f2cb39701927\") " pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:23.243352 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.243312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:23.275327 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.275293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:23.517101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.517077    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-64lcz"]
Sep 09 20:05:23.518512 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:05:23.518484    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b780fd5_3f70_460d_a236_ce244b4359d5.slice/crio-de26ba0e1e50c914c91bd20ce6889d18e282c9e40b1e97558aa20b1f827595e7 WatchSource:0}: Error finding container de26ba0e1e50c914c91bd20ce6889d18e282c9e40b1e97558aa20b1f827595e7: Status 404 returned error can't find the container with id de26ba0e1e50c914c91bd20ce6889d18e282c9e40b1e97558aa20b1f827595e7
Sep 09 20:05:23.555732 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.555711    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"]
Sep 09 20:05:23.557859 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:05:23.557827    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod92a1c2ba_add6_4467_877a_f2cb39701927.slice/crio-b4e390778f844c82c3c6aa68f999aa9e12c97168c43cb8baede0c7439bd4b833 WatchSource:0}: Error finding container b4e390778f844c82c3c6aa68f999aa9e12c97168c43cb8baede0c7439bd4b833: Status 404 returned error can't find the container with id b4e390778f844c82c3c6aa68f999aa9e12c97168c43cb8baede0c7439bd4b833
Sep 09 20:05:23.621827 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.621790    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-64lcz" event={"ID":"0b780fd5-3f70-460d-a236-ce244b4359d5","Type":"ContainerStarted","Data":"de26ba0e1e50c914c91bd20ce6889d18e282c9e40b1e97558aa20b1f827595e7"}
Sep 09 20:05:23.622863 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:23.622842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd" event={"ID":"92a1c2ba-add6-4467-877a-f2cb39701927","Type":"ContainerStarted","Data":"b4e390778f844c82c3c6aa68f999aa9e12c97168c43cb8baede0c7439bd4b833"}
Sep 09 20:05:27.647516 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:27.647473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-64lcz" event={"ID":"0b780fd5-3f70-460d-a236-ce244b4359d5","Type":"ContainerStarted","Data":"0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e"}
Sep 09 20:05:27.647985 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:27.647538    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:05:27.649094 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:27.649073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd" event={"ID":"92a1c2ba-add6-4467-877a-f2cb39701927","Type":"ContainerStarted","Data":"3207296b44492e6b1b3f18091d36d6c30d7f6405ea3089a12a771527e4939de1"}
Sep 09 20:05:27.649184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:27.649128    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:27.669512 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:27.669471    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7676c798d9-64lcz" podStartSLOduration=2.3004904010000002 podStartE2EDuration="5.66944878s" totalImagesPullingTime="3.368958379s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:27.667058947 +0000 UTC m=+630.185196474" watchObservedRunningTime="2026-09-09 20:05:27.66944878 +0000 UTC m=+630.187586339"
Sep 09 20:05:27.688503 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:27.688457    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd" podStartSLOduration=2.355343506 podStartE2EDuration="5.688446263s" totalImagesPullingTime="3.333102757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:05:27.688212064 +0000 UTC m=+630.206349582" watchObservedRunningTime="2026-09-09 20:05:27.688446263 +0000 UTC m=+630.206583790"
Sep 09 20:05:58.656176 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:58.656146    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-59449d9fbd-kl5hd"
Sep 09 20:05:58.659162 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:05:58.659140    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:06:08.405685 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.405649    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-64lcz"]
Sep 09 20:06:08.406206 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.405828    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-7676c798d9-64lcz" podUID="0b780fd5-3f70-460d-a236-ce244b4359d5" containerName="manager" containerID="cri-o://0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e" gracePeriod=10
Sep 09 20:06:08.441608 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.441582    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-sq6lc"]
Sep 09 20:06:08.443561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.443546    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.453802 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.453780    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-sq6lc"]
Sep 09 20:06:08.551407 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.551370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvx2z\" (UniqueName: \"kubernetes.io/projected/7a6535f4-825c-4d3b-b10c-fd61b6f9ab58-kube-api-access-lvx2z\") pod \"kserve-controller-manager-7676c798d9-sq6lc\" (UID: \"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58\") " pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.551576 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.551485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a6535f4-825c-4d3b-b10c-fd61b6f9ab58-cert\") pod \"kserve-controller-manager-7676c798d9-sq6lc\" (UID: \"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58\") " pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.642837 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.642813    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:06:08.652051 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.652022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a6535f4-825c-4d3b-b10c-fd61b6f9ab58-cert\") pod \"kserve-controller-manager-7676c798d9-sq6lc\" (UID: \"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58\") " pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.652135 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.652076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvx2z\" (UniqueName: \"kubernetes.io/projected/7a6535f4-825c-4d3b-b10c-fd61b6f9ab58-kube-api-access-lvx2z\") pod \"kserve-controller-manager-7676c798d9-sq6lc\" (UID: \"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58\") " pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.654401 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.654376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7a6535f4-825c-4d3b-b10c-fd61b6f9ab58-cert\") pod \"kserve-controller-manager-7676c798d9-sq6lc\" (UID: \"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58\") " pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.665223 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.665165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvx2z\" (UniqueName: \"kubernetes.io/projected/7a6535f4-825c-4d3b-b10c-fd61b6f9ab58-kube-api-access-lvx2z\") pod \"kserve-controller-manager-7676c798d9-sq6lc\" (UID: \"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58\") " pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.753110 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.753080    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert\") pod \"0b780fd5-3f70-460d-a236-ce244b4359d5\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") "
Sep 09 20:06:08.753306 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.753120    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk649\" (UniqueName: \"kubernetes.io/projected/0b780fd5-3f70-460d-a236-ce244b4359d5-kube-api-access-fk649\") pod \"0b780fd5-3f70-460d-a236-ce244b4359d5\" (UID: \"0b780fd5-3f70-460d-a236-ce244b4359d5\") "
Sep 09 20:06:08.755388 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.755356    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert" (OuterVolumeSpecName: "cert") pod "0b780fd5-3f70-460d-a236-ce244b4359d5" (UID: "0b780fd5-3f70-460d-a236-ce244b4359d5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:06:08.755499 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.755424    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b780fd5-3f70-460d-a236-ce244b4359d5-kube-api-access-fk649" (OuterVolumeSpecName: "kube-api-access-fk649") pod "0b780fd5-3f70-460d-a236-ce244b4359d5" (UID: "0b780fd5-3f70-460d-a236-ce244b4359d5"). InnerVolumeSpecName "kube-api-access-fk649". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:06:08.802922 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.802895    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:08.846006 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.845973    2576 generic.go:356] "Generic (PLEG): container finished" podID="0b780fd5-3f70-460d-a236-ce244b4359d5" containerID="0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e" exitCode=0
Sep 09 20:06:08.846172 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.846012    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-7676c798d9-64lcz"
Sep 09 20:06:08.846172 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.846027    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-64lcz" event={"ID":"0b780fd5-3f70-460d-a236-ce244b4359d5","Type":"ContainerDied","Data":"0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e"}
Sep 09 20:06:08.846172 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.846070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-64lcz" event={"ID":"0b780fd5-3f70-460d-a236-ce244b4359d5","Type":"ContainerDied","Data":"de26ba0e1e50c914c91bd20ce6889d18e282c9e40b1e97558aa20b1f827595e7"}
Sep 09 20:06:08.846172 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.846090    2576 scope.go:117] "RemoveContainer" containerID="0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e"
Sep 09 20:06:08.854245 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.854206    2576 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b780fd5-3f70-460d-a236-ce244b4359d5-cert\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:06:08.854245 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.854235    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk649\" (UniqueName: \"kubernetes.io/projected/0b780fd5-3f70-460d-a236-ce244b4359d5-kube-api-access-fk649\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:06:08.859113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.859085    2576 scope.go:117] "RemoveContainer" containerID="0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e"
Sep 09 20:06:08.859480 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:06:08.859456    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e\": container with ID starting with 0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e not found: ID does not exist" containerID="0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e"
Sep 09 20:06:08.859572 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.859491    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e"} err="failed to get container status \"0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e\": rpc error: code = NotFound desc = could not find container \"0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e\": container with ID starting with 0b5996640112eb4ef3f33b26e2988b2fdbe5132beec108c34f35e1d8c4a0576e not found: ID does not exist"
Sep 09 20:06:08.876638 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.876577    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-64lcz"]
Sep 09 20:06:08.879673 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:08.879647    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-64lcz"]
Sep 09 20:06:09.082890 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.082864    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-7676c798d9-sq6lc"]
Sep 09 20:06:09.084411 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:06:09.084385    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a6535f4_825c_4d3b_b10c_fd61b6f9ab58.slice/crio-205be258f3af72bea4c5f9c57bc2bd0e5be9fa118516d918537d629c35058682 WatchSource:0}: Error finding container 205be258f3af72bea4c5f9c57bc2bd0e5be9fa118516d918537d629c35058682: Status 404 returned error can't find the container with id 205be258f3af72bea4c5f9c57bc2bd0e5be9fa118516d918537d629c35058682
Sep 09 20:06:09.085709 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.085693    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:06:09.851923 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.851887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc" event={"ID":"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58","Type":"ContainerStarted","Data":"13b15ca7016e99525f7ac34602b92c51bc25c6e149394cc0879c133a3b46f7fa"}
Sep 09 20:06:09.852349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.851948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc" event={"ID":"7a6535f4-825c-4d3b-b10c-fd61b6f9ab58","Type":"ContainerStarted","Data":"205be258f3af72bea4c5f9c57bc2bd0e5be9fa118516d918537d629c35058682"}
Sep 09 20:06:09.852349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.852001    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:06:09.871982 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.871908    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc" podStartSLOduration=1.333120398 podStartE2EDuration="1.871892608s" totalImagesPullingTime="538.77221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:06:09.870077324 +0000 UTC m=+672.388214852" watchObservedRunningTime="2026-09-09 20:06:09.871892608 +0000 UTC m=+672.390030135"
Sep 09 20:06:09.961761 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:09.961727    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b780fd5-3f70-460d-a236-ce244b4359d5" path="/var/lib/kubelet/pods/0b780fd5-3f70-460d-a236-ce244b4359d5/volumes"
Sep 09 20:06:40.863283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:06:40.863250    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-7676c798d9-sq6lc"
Sep 09 20:08:40.430011 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.429922    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"]
Sep 09 20:08:40.430436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.430241    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b780fd5-3f70-460d-a236-ce244b4359d5" containerName="manager"
Sep 09 20:08:40.430436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.430251    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b780fd5-3f70-460d-a236-ce244b4359d5" containerName="manager"
Sep 09 20:08:40.430436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.430310    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b780fd5-3f70-460d-a236-ce244b4359d5" containerName="manager"
Sep 09 20:08:40.433121 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.433101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.435817 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.435797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 09 20:08:40.436785 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.436767    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-epp-sa-dockercfg-jfm4d\""
Sep 09 20:08:40.437004 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.436986    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 09 20:08:40.437086 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.437030    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 09 20:08:40.437146 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.437109    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-4m7n8\""
Sep 09 20:08:40.442355 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.442332    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"]
Sep 09 20:08:40.572809 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.572776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rztvm\" (UniqueName: \"kubernetes.io/projected/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-kube-api-access-rztvm\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.573016 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.572831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.673755 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.673721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rztvm\" (UniqueName: \"kubernetes.io/projected/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-kube-api-access-rztvm\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.673937 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.673775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.676469 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.676443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-tls-certs\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.682651 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.682600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rztvm\" (UniqueName: \"kubernetes.io/projected/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-kube-api-access-rztvm\") pod \"fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:40.744147 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:40.744109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:41.017614 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:41.017587    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"]
Sep 09 20:08:41.020569 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:08:41.020537    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68a5ed1_950a_48d4_a0cc_48b67a0f7fbc.slice/crio-3d876e0d51079109b754aa54620e0c9f49a8ecc9d5de16e8cfbb61ffef0e3d5b WatchSource:0}: Error finding container 3d876e0d51079109b754aa54620e0c9f49a8ecc9d5de16e8cfbb61ffef0e3d5b: Status 404 returned error can't find the container with id 3d876e0d51079109b754aa54620e0c9f49a8ecc9d5de16e8cfbb61ffef0e3d5b
Sep 09 20:08:41.588874 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:41.588837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" event={"ID":"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc","Type":"ContainerStarted","Data":"3d876e0d51079109b754aa54620e0c9f49a8ecc9d5de16e8cfbb61ffef0e3d5b"}
Sep 09 20:08:44.606491 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:44.606458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" event={"ID":"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc","Type":"ContainerStarted","Data":"ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8"}
Sep 09 20:08:44.606851 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:44.606583    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:08:44.625847 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:08:44.625797    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podStartSLOduration=2.064436504 podStartE2EDuration="4.625784135s" totalImagesPullingTime="2.561347631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:08:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:08:44.624694155 +0000 UTC m=+827.142831683" watchObservedRunningTime="2026-09-09 20:08:44.625784135 +0000 UTC m=+827.143921663"
Sep 09 20:09:15.615097 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:15.615062    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:09:22.459231 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:22.459192    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"]
Sep 09 20:09:25.615208 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:25.615165    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:09:35.614345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:35.614300    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:09:37.484287 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:37.484244    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main" containerID="cri-o://ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8" gracePeriod=45
Sep 09 20:09:45.615030 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:45.614989    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:09:45.615426 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:45.615091    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:09:55.614801 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:55.614761    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:09:57.930024 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:57.929995    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:09:57.935613 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:09:57.935580    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:10:05.615055 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:05.615016    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:10:07.739791 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.739765    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:10:07.857365 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.857336    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rztvm\" (UniqueName: \"kubernetes.io/projected/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-kube-api-access-rztvm\") pod \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") "
Sep 09 20:10:07.857548 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.857447    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-tls-certs\") pod \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\" (UID: \"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc\") "
Sep 09 20:10:07.859699 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.859671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-kube-api-access-rztvm" (OuterVolumeSpecName: "kube-api-access-rztvm") pod "b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" (UID: "b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc"). InnerVolumeSpecName "kube-api-access-rztvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:10:07.859699 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.859688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" (UID: "b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:10:07.958415 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.958382    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:10:07.958415 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:07.958410    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rztvm\" (UniqueName: \"kubernetes.io/projected/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc-kube-api-access-rztvm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:10:08.014077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.014043    2576 generic.go:356] "Generic (PLEG): container finished" podID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerID="ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8" exitCode=0
Sep 09 20:10:08.014227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.014087    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"
Sep 09 20:10:08.014227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.014099    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" event={"ID":"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc","Type":"ContainerDied","Data":"ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8"}
Sep 09 20:10:08.014227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.014154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q" event={"ID":"b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc","Type":"ContainerDied","Data":"3d876e0d51079109b754aa54620e0c9f49a8ecc9d5de16e8cfbb61ffef0e3d5b"}
Sep 09 20:10:08.014227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.014178    2576 scope.go:117] "RemoveContainer" containerID="ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8"
Sep 09 20:10:08.022516 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.022497    2576 scope.go:117] "RemoveContainer" containerID="ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8"
Sep 09 20:10:08.022765 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:10:08.022747    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8\": container with ID starting with ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8 not found: ID does not exist" containerID="ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8"
Sep 09 20:10:08.022825 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.022773    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8"} err="failed to get container status \"ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8\": rpc error: code = NotFound desc = could not find container \"ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8\": container with ID starting with ed92b94f32016a0749a2ae619ff78318433f7459e295811726ddcafdbcbf44e8 not found: ID does not exist"
Sep 09 20:10:08.036301 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.036278    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"]
Sep 09 20:10:08.040384 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:08.040362    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-router-scheduler-76d5658f88-szl9q"]
Sep 09 20:10:09.962181 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:09.962147    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" path="/var/lib/kubelet/pods/b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc/volumes"
Sep 09 20:10:26.644086 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.644051    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"]
Sep 09 20:10:26.644677 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.644361    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main"
Sep 09 20:10:26.644677 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.644371    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main"
Sep 09 20:10:26.644677 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.644417    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b68a5ed1-950a-48d4-a0cc-48b67a0f7fbc" containerName="main"
Sep 09 20:10:26.646252 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.646223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.650217 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.650153    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-sfpck\""
Sep 09 20:10:26.650217 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.650172    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 09 20:10:26.650418 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.650172    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 09 20:10:26.650418 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.650153    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-epp-sa-dockercfg-bqg56\""
Sep 09 20:10:26.650418 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.650362    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 09 20:10:26.657144 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.657121    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"]
Sep 09 20:10:26.817458 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.817419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08101017-1441-41ca-b4c9-d63c236afe72-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.817655 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.817507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr95d\" (UniqueName: \"kubernetes.io/projected/08101017-1441-41ca-b4c9-d63c236afe72-kube-api-access-cr95d\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.919007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.918884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr95d\" (UniqueName: \"kubernetes.io/projected/08101017-1441-41ca-b4c9-d63c236afe72-kube-api-access-cr95d\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.919007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.918943    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08101017-1441-41ca-b4c9-d63c236afe72-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.921648 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.921619    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08101017-1441-41ca-b4c9-d63c236afe72-tls-certs\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.930547 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.930523    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr95d\" (UniqueName: \"kubernetes.io/projected/08101017-1441-41ca-b4c9-d63c236afe72-kube-api-access-cr95d\") pod \"fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:26.958901 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:26.958873    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:27.232911 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:27.232809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"]
Sep 09 20:10:27.235910 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:10:27.235884    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08101017_1441_41ca_b4c9_d63c236afe72.slice/crio-7339a5c128bf8ddd16f50d7b5cf5367b3d5a953c88730d9441f3d6861e007cd6 WatchSource:0}: Error finding container 7339a5c128bf8ddd16f50d7b5cf5367b3d5a953c88730d9441f3d6861e007cd6: Status 404 returned error can't find the container with id 7339a5c128bf8ddd16f50d7b5cf5367b3d5a953c88730d9441f3d6861e007cd6
Sep 09 20:10:28.111680 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:28.111645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" event={"ID":"08101017-1441-41ca-b4c9-d63c236afe72","Type":"ContainerStarted","Data":"69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663"}
Sep 09 20:10:28.112093 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:28.111697    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" event={"ID":"08101017-1441-41ca-b4c9-d63c236afe72","Type":"ContainerStarted","Data":"7339a5c128bf8ddd16f50d7b5cf5367b3d5a953c88730d9441f3d6861e007cd6"}
Sep 09 20:10:28.112093 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:28.111749    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:10:28.132794 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:28.132738    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podStartSLOduration=2.132721304 podStartE2EDuration="2.132721304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:10:28.130256406 +0000 UTC m=+930.648393956" watchObservedRunningTime="2026-09-09 20:10:28.132721304 +0000 UTC m=+930.650858831"
Sep 09 20:10:59.118106 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:10:59.118072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:11:05.930527 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:05.930497    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"]
Sep 09 20:11:09.118922 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:09.118887    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:11:19.118273 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:19.118191    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:11:20.950936 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:20.950894    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main" containerID="cri-o://69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663" gracePeriod=45
Sep 09 20:11:29.118877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:29.118833    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:11:29.119372 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:29.118932    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:11:39.118152 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:39.118113    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:11:49.118829 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:49.118790    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:11:51.188914 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.188891    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:11:51.201994 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.201937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08101017-1441-41ca-b4c9-d63c236afe72-tls-certs\") pod \"08101017-1441-41ca-b4c9-d63c236afe72\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") "
Sep 09 20:11:51.204155 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.204133    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08101017-1441-41ca-b4c9-d63c236afe72-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "08101017-1441-41ca-b4c9-d63c236afe72" (UID: "08101017-1441-41ca-b4c9-d63c236afe72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:11:51.302991 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.302935    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cr95d\" (UniqueName: \"kubernetes.io/projected/08101017-1441-41ca-b4c9-d63c236afe72-kube-api-access-cr95d\") pod \"08101017-1441-41ca-b4c9-d63c236afe72\" (UID: \"08101017-1441-41ca-b4c9-d63c236afe72\") "
Sep 09 20:11:51.303159 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.303091    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08101017-1441-41ca-b4c9-d63c236afe72-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:11:51.305167 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.305140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08101017-1441-41ca-b4c9-d63c236afe72-kube-api-access-cr95d" (OuterVolumeSpecName: "kube-api-access-cr95d") pod "08101017-1441-41ca-b4c9-d63c236afe72" (UID: "08101017-1441-41ca-b4c9-d63c236afe72"). InnerVolumeSpecName "kube-api-access-cr95d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:11:51.403450 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.403413    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr95d\" (UniqueName: \"kubernetes.io/projected/08101017-1441-41ca-b4c9-d63c236afe72-kube-api-access-cr95d\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:11:51.522462 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.522377    2576 generic.go:356] "Generic (PLEG): container finished" podID="08101017-1441-41ca-b4c9-d63c236afe72" containerID="69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663" exitCode=0
Sep 09 20:11:51.522462 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.522418    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"
Sep 09 20:11:51.522640 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.522459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" event={"ID":"08101017-1441-41ca-b4c9-d63c236afe72","Type":"ContainerDied","Data":"69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663"}
Sep 09 20:11:51.522640 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.522508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28" event={"ID":"08101017-1441-41ca-b4c9-d63c236afe72","Type":"ContainerDied","Data":"7339a5c128bf8ddd16f50d7b5cf5367b3d5a953c88730d9441f3d6861e007cd6"}
Sep 09 20:11:51.522640 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.522531    2576 scope.go:117] "RemoveContainer" containerID="69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663"
Sep 09 20:11:51.531396 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.531378    2576 scope.go:117] "RemoveContainer" containerID="69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663"
Sep 09 20:11:51.531706 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:11:51.531686    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663\": container with ID starting with 69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663 not found: ID does not exist" containerID="69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663"
Sep 09 20:11:51.531759 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.531713    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663"} err="failed to get container status \"69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663\": rpc error: code = NotFound desc = could not find container \"69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663\": container with ID starting with 69b8fde856ec93ca59ad780ad7493eebc685fe2bf42d1b11bad9f9d219a97663 not found: ID does not exist"
Sep 09 20:11:51.544155 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.544130    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"]
Sep 09 20:11:51.547511 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.547491    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-router-scheduler-88dbd48f9-t2s28"]
Sep 09 20:11:51.961397 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:11:51.961365    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08101017-1441-41ca-b4c9-d63c236afe72" path="/var/lib/kubelet/pods/08101017-1441-41ca-b4c9-d63c236afe72/volumes"
Sep 09 20:12:24.175095 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.175058    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"]
Sep 09 20:12:24.175481 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.175414    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main"
Sep 09 20:12:24.175481 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.175425    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main"
Sep 09 20:12:24.175481 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.175479    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="08101017-1441-41ca-b4c9-d63c236afe72" containerName="main"
Sep 09 20:12:24.178445 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.178422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.181359 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.181333    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 09 20:12:24.182288 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.182261    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 09 20:12:24.182413 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.182286    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 09 20:12:24.182413 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.182385    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-t9xcm\""
Sep 09 20:12:24.182638 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.182623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-kwgcw\""
Sep 09 20:12:24.188154 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.188131    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"]
Sep 09 20:12:24.261478 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.261439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.261635 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.261556    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmsv2\" (UniqueName: \"kubernetes.io/projected/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-kube-api-access-lmsv2\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.362732 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.362698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.362930 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.362803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmsv2\" (UniqueName: \"kubernetes.io/projected/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-kube-api-access-lmsv2\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.365560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.365520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.370933 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.370905    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmsv2\" (UniqueName: \"kubernetes.io/projected/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-kube-api-access-lmsv2\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.489130 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.489020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:24.762397 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:24.762368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"]
Sep 09 20:12:24.764440 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:12:24.764410    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1e4bc5d_3fab_46e4_af1d_dee6f4a3a3a4.slice/crio-6109b87aab1d95a05572ee903c358872e3e03ead228021322afbdb064f376afb WatchSource:0}: Error finding container 6109b87aab1d95a05572ee903c358872e3e03ead228021322afbdb064f376afb: Status 404 returned error can't find the container with id 6109b87aab1d95a05572ee903c358872e3e03ead228021322afbdb064f376afb
Sep 09 20:12:25.689528 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:25.689489    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" event={"ID":"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4","Type":"ContainerStarted","Data":"6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e"}
Sep 09 20:12:25.689945 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:25.689563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:12:25.689945 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:25.689587    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" event={"ID":"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4","Type":"ContainerStarted","Data":"6109b87aab1d95a05572ee903c358872e3e03ead228021322afbdb064f376afb"}
Sep 09 20:12:25.710010 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:25.709946    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" podStartSLOduration=1.709930044 podStartE2EDuration="1.709930044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:12:25.708110746 +0000 UTC m=+1048.226248287" watchObservedRunningTime="2026-09-09 20:12:25.709930044 +0000 UTC m=+1048.228067571"
Sep 09 20:12:56.697124 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:12:56.697094    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:13:13.022769 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.022733    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"]
Sep 09 20:13:13.026184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.026165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.029801 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.029778    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 09 20:13:13.029934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.029824    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-td82v\""
Sep 09 20:13:13.029934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.029834    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 09 20:13:13.029934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.029870    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-rtqv4\""
Sep 09 20:13:13.029934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.029886    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 09 20:13:13.034982 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.034939    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"]
Sep 09 20:13:13.053807 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.053775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.053990 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.053849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h96x\" (UniqueName: \"kubernetes.io/projected/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-kube-api-access-7h96x\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.154252 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.154217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h96x\" (UniqueName: \"kubernetes.io/projected/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-kube-api-access-7h96x\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.154415 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.154291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.156877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.156858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.162734 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.162709    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h96x\" (UniqueName: \"kubernetes.io/projected/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-kube-api-access-7h96x\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.338909 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.338824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.614008 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.613980    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"]
Sep 09 20:13:13.615536 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:13:13.615507    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2156e7_e29c_4526_8ea8_fbffbf9ee522.slice/crio-6e44e239f3d95c68a62b38bcddf178249d9e6964355b9401ee65dd91f441a50b WatchSource:0}: Error finding container 6e44e239f3d95c68a62b38bcddf178249d9e6964355b9401ee65dd91f441a50b: Status 404 returned error can't find the container with id 6e44e239f3d95c68a62b38bcddf178249d9e6964355b9401ee65dd91f441a50b
Sep 09 20:13:13.928424 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.928328    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" event={"ID":"6f2156e7-e29c-4526-8ea8-fbffbf9ee522","Type":"ContainerStarted","Data":"f60d0d49fa5de90226a82b8f684add850cfca5e1be90fd20e0d58d2be6521aa0"}
Sep 09 20:13:13.928424 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.928392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" event={"ID":"6f2156e7-e29c-4526-8ea8-fbffbf9ee522","Type":"ContainerStarted","Data":"6e44e239f3d95c68a62b38bcddf178249d9e6964355b9401ee65dd91f441a50b"}
Sep 09 20:13:13.928424 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.928415    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:13:13.943879 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:13.943832    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podStartSLOduration=0.943816701 podStartE2EDuration="943.816701ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:13:13.943505132 +0000 UTC m=+1096.461642659" watchObservedRunningTime="2026-09-09 20:13:13.943816701 +0000 UTC m=+1096.461954227"
Sep 09 20:13:44.934870 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:13:44.934840    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:14:20.680448 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:20.680361    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"]
Sep 09 20:14:26.697385 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:26.697347    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:14:35.701803 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:35.701759    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main" containerID="cri-o://6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e" gracePeriod=45
Sep 09 20:14:36.697046 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:36.697008    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:14:46.696841 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:46.696797    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:14:46.697391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:46.696921    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:14:56.696808 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:56.696766    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:14:57.968418 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:57.968391    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:14:57.970664 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:14:57.970644    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:15:05.959753 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:05.959724    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:15:06.081938 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.081839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-tls-certs\") pod \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") "
Sep 09 20:15:06.081938 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.081910    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmsv2\" (UniqueName: \"kubernetes.io/projected/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-kube-api-access-lmsv2\") pod \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\" (UID: \"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4\") "
Sep 09 20:15:06.084360 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.084331    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" (UID: "f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:15:06.084360 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.084337    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-kube-api-access-lmsv2" (OuterVolumeSpecName: "kube-api-access-lmsv2") pod "f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" (UID: "f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4"). InnerVolumeSpecName "kube-api-access-lmsv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:15:06.182941 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.182901    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:15:06.182941 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.182932    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmsv2\" (UniqueName: \"kubernetes.io/projected/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4-kube-api-access-lmsv2\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:15:06.483092 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.483054    2576 generic.go:356] "Generic (PLEG): container finished" podID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerID="6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e" exitCode=0
Sep 09 20:15:06.483287 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.483100    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"
Sep 09 20:15:06.483287 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.483144    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" event={"ID":"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4","Type":"ContainerDied","Data":"6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e"}
Sep 09 20:15:06.483287 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.483191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6" event={"ID":"f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4","Type":"ContainerDied","Data":"6109b87aab1d95a05572ee903c358872e3e03ead228021322afbdb064f376afb"}
Sep 09 20:15:06.483287 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.483214    2576 scope.go:117] "RemoveContainer" containerID="6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e"
Sep 09 20:15:06.492491 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.492472    2576 scope.go:117] "RemoveContainer" containerID="6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e"
Sep 09 20:15:06.492767 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:15:06.492747    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e\": container with ID starting with 6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e not found: ID does not exist" containerID="6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e"
Sep 09 20:15:06.492802 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.492776    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e"} err="failed to get container status \"6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e\": rpc error: code = NotFound desc = could not find container \"6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e\": container with ID starting with 6dea340b7c06217da2d231071cd6a037c0eba432b7d8a86ca3cbf7b6ef701a1e not found: ID does not exist"
Sep 09 20:15:06.504587 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.504557    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"]
Sep 09 20:15:06.506507 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:06.506484    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-sche6zlb6"]
Sep 09 20:15:07.962361 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:07.962325    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" path="/var/lib/kubelet/pods/f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4/volumes"
Sep 09 20:15:16.157475 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.157433    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"]
Sep 09 20:15:16.158060 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.157801    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main"
Sep 09 20:15:16.158060 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.157817    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main"
Sep 09 20:15:16.158060 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.157934    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1e4bc5d-3fab-46e4-af1d-dee6f4a3a3a4" containerName="main"
Sep 09 20:15:16.162747 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.162719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.165661 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.165632    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 09 20:15:16.166651 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.166623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-krtpk\""
Sep 09 20:15:16.166651 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.166658    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 09 20:15:16.166871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.166853    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 09 20:15:16.167103 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.166889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-epp-sa-dockercfg-xtpmp\""
Sep 09 20:15:16.172676 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.172655    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"]
Sep 09 20:15:16.263415 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.263378    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psfbj\" (UniqueName: \"kubernetes.io/projected/5dcb6ea4-7965-4235-ab90-15b2e1272d55-kube-api-access-psfbj\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.263596 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.263428    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcb6ea4-7965-4235-ab90-15b2e1272d55-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.364806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.364769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psfbj\" (UniqueName: \"kubernetes.io/projected/5dcb6ea4-7965-4235-ab90-15b2e1272d55-kube-api-access-psfbj\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.364806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.364814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcb6ea4-7965-4235-ab90-15b2e1272d55-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.367736 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.367697    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcb6ea4-7965-4235-ab90-15b2e1272d55-tls-certs\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.372881 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.372857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psfbj\" (UniqueName: \"kubernetes.io/projected/5dcb6ea4-7965-4235-ab90-15b2e1272d55-kube-api-access-psfbj\") pod \"scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.477663 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.477573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:16.752218 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:16.752175    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"]
Sep 09 20:15:16.753440 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:15:16.753414    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcb6ea4_7965_4235_ab90_15b2e1272d55.slice/crio-d153e6b738751d44006e0109fd081e7f1e058d06bb79b041f1749537834e6106 WatchSource:0}: Error finding container d153e6b738751d44006e0109fd081e7f1e058d06bb79b041f1749537834e6106: Status 404 returned error can't find the container with id d153e6b738751d44006e0109fd081e7f1e058d06bb79b041f1749537834e6106
Sep 09 20:15:17.541268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:17.541237    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" event={"ID":"5dcb6ea4-7965-4235-ab90-15b2e1272d55","Type":"ContainerStarted","Data":"51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10"}
Sep 09 20:15:17.541696 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:17.541288    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" event={"ID":"5dcb6ea4-7965-4235-ab90-15b2e1272d55","Type":"ContainerStarted","Data":"d153e6b738751d44006e0109fd081e7f1e058d06bb79b041f1749537834e6106"}
Sep 09 20:15:17.541696 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:17.541327    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:17.559151 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:17.559101    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" podStartSLOduration=1.559079495 podStartE2EDuration="1.559079495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:15:17.557935096 +0000 UTC m=+1220.076072692" watchObservedRunningTime="2026-09-09 20:15:17.559079495 +0000 UTC m=+1220.077217021"
Sep 09 20:15:32.586113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:32.586080    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"]
Sep 09 20:15:34.935806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:34.935764    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:15:44.935103 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:44.935056    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:15:47.608313 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:47.608247    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main" containerID="cri-o://f60d0d49fa5de90226a82b8f684add850cfca5e1be90fd20e0d58d2be6521aa0" gracePeriod=45
Sep 09 20:15:48.548487 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:48.548454    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:15:50.605575 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:50.605543    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"]
Sep 09 20:15:54.934627 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:54.934578    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:15:54.935019 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:54.934680    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:15:58.548608 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:15:58.548550    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:16:04.934727 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:04.934689    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:16:05.625797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:05.625736    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main" containerID="cri-o://51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10" gracePeriod=45
Sep 09 20:16:08.548298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:08.548261    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:16:14.934418 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:14.934380    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:16:17.842039 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.842008    2576 generic.go:356] "Generic (PLEG): container finished" podID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerID="f60d0d49fa5de90226a82b8f684add850cfca5e1be90fd20e0d58d2be6521aa0" exitCode=0
Sep 09 20:16:17.842427 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.842061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" event={"ID":"6f2156e7-e29c-4526-8ea8-fbffbf9ee522","Type":"ContainerDied","Data":"f60d0d49fa5de90226a82b8f684add850cfca5e1be90fd20e0d58d2be6521aa0"}
Sep 09 20:16:17.861407 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.861386    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:16:17.919400 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.919333    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7h96x\" (UniqueName: \"kubernetes.io/projected/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-kube-api-access-7h96x\") pod \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") "
Sep 09 20:16:17.919653 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.919419    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-tls-certs\") pod \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\" (UID: \"6f2156e7-e29c-4526-8ea8-fbffbf9ee522\") "
Sep 09 20:16:17.921510 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.921483    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-kube-api-access-7h96x" (OuterVolumeSpecName: "kube-api-access-7h96x") pod "6f2156e7-e29c-4526-8ea8-fbffbf9ee522" (UID: "6f2156e7-e29c-4526-8ea8-fbffbf9ee522"). InnerVolumeSpecName "kube-api-access-7h96x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:17.921606 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:17.921593    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f2156e7-e29c-4526-8ea8-fbffbf9ee522" (UID: "6f2156e7-e29c-4526-8ea8-fbffbf9ee522"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:16:18.020180 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.020143    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:16:18.020180 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.020175    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7h96x\" (UniqueName: \"kubernetes.io/projected/6f2156e7-e29c-4526-8ea8-fbffbf9ee522-kube-api-access-7h96x\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:16:18.548762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.548720    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:16:18.548983 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.548837    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:16:18.847730 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.847699    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"
Sep 09 20:16:18.848184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.847725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z" event={"ID":"6f2156e7-e29c-4526-8ea8-fbffbf9ee522","Type":"ContainerDied","Data":"6e44e239f3d95c68a62b38bcddf178249d9e6964355b9401ee65dd91f441a50b"}
Sep 09 20:16:18.848184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.847808    2576 scope.go:117] "RemoveContainer" containerID="f60d0d49fa5de90226a82b8f684add850cfca5e1be90fd20e0d58d2be6521aa0"
Sep 09 20:16:18.865799 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.865774    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"]
Sep 09 20:16:18.870366 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:18.870343    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-9mj6z"]
Sep 09 20:16:19.961765 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:19.961732    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" path="/var/lib/kubelet/pods/6f2156e7-e29c-4526-8ea8-fbffbf9ee522/volumes"
Sep 09 20:16:28.548046 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:28.548010    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:16:35.618227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.618193    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"]
Sep 09 20:16:35.618686 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.618535    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main"
Sep 09 20:16:35.618686 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.618547    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main"
Sep 09 20:16:35.618686 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.618613    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2156e7-e29c-4526-8ea8-fbffbf9ee522" containerName="main"
Sep 09 20:16:35.621685 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.621664    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.625559 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.625524    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"default-dockercfg-9xpg9\""
Sep 09 20:16:35.625559 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.625545    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"openshift-service-ca.crt\""
Sep 09 20:16:35.625774 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.625594    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-epp-sa-dockercfg-q6s28\""
Sep 09 20:16:35.625774 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.625665    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"kube-root-ca.crt\""
Sep 09 20:16:35.625896 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.625879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Sep 09 20:16:35.636605 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.636581    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"]
Sep 09 20:16:35.773500 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.773459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cldz\" (UniqueName: \"kubernetes.io/projected/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-kube-api-access-9cldz\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.773657 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.773523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.874838 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.874764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cldz\" (UniqueName: \"kubernetes.io/projected/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-kube-api-access-9cldz\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.874976 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.874834    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.877460 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.877426    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.881420 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.881403    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:16:35.884643 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.884623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cldz\" (UniqueName: \"kubernetes.io/projected/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-kube-api-access-9cldz\") pod \"auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") " pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.932728 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.932685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:35.934092 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.934064    2576 generic.go:356] "Generic (PLEG): container finished" podID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerID="51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10" exitCode=0
Sep 09 20:16:35.934222 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.934098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" event={"ID":"5dcb6ea4-7965-4235-ab90-15b2e1272d55","Type":"ContainerDied","Data":"51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10"}
Sep 09 20:16:35.934222 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.934162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z" event={"ID":"5dcb6ea4-7965-4235-ab90-15b2e1272d55","Type":"ContainerDied","Data":"d153e6b738751d44006e0109fd081e7f1e058d06bb79b041f1749537834e6106"}
Sep 09 20:16:35.934222 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.934110    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"
Sep 09 20:16:35.934222 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.934200    2576 scope.go:117] "RemoveContainer" containerID="51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10"
Sep 09 20:16:35.943196 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.943172    2576 scope.go:117] "RemoveContainer" containerID="51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10"
Sep 09 20:16:35.943464 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:16:35.943446    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10\": container with ID starting with 51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10 not found: ID does not exist" containerID="51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10"
Sep 09 20:16:35.943559 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.943469    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10"} err="failed to get container status \"51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10\": rpc error: code = NotFound desc = could not find container \"51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10\": container with ID starting with 51f07b6db5b7b6ec6d54178b35982ee44b6cdc3c90afa960feaa4b587c787b10 not found: ID does not exist"
Sep 09 20:16:35.976551 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.975904    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcb6ea4-7965-4235-ab90-15b2e1272d55-tls-certs\") pod \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") "
Sep 09 20:16:35.976551 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.976363    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psfbj\" (UniqueName: \"kubernetes.io/projected/5dcb6ea4-7965-4235-ab90-15b2e1272d55-kube-api-access-psfbj\") pod \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\" (UID: \"5dcb6ea4-7965-4235-ab90-15b2e1272d55\") "
Sep 09 20:16:35.978466 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.978440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dcb6ea4-7965-4235-ab90-15b2e1272d55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dcb6ea4-7965-4235-ab90-15b2e1272d55" (UID: "5dcb6ea4-7965-4235-ab90-15b2e1272d55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:16:35.978712 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:35.978690    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcb6ea4-7965-4235-ab90-15b2e1272d55-kube-api-access-psfbj" (OuterVolumeSpecName: "kube-api-access-psfbj") pod "5dcb6ea4-7965-4235-ab90-15b2e1272d55" (UID: "5dcb6ea4-7965-4235-ab90-15b2e1272d55"). InnerVolumeSpecName "kube-api-access-psfbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:16:36.077833 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.077633    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dcb6ea4-7965-4235-ab90-15b2e1272d55-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:16:36.077833 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.077661    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psfbj\" (UniqueName: \"kubernetes.io/projected/5dcb6ea4-7965-4235-ab90-15b2e1272d55-kube-api-access-psfbj\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:16:36.224676 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.224627    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"]
Sep 09 20:16:36.226947 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:16:36.226918    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac95f3d_73e4_4e05_86f7_c0ec4f49c74d.slice/crio-0e90bd2953ec6dfa4104bc47e99ce28a4543898f12a087252eddae730da3afed WatchSource:0}: Error finding container 0e90bd2953ec6dfa4104bc47e99ce28a4543898f12a087252eddae730da3afed: Status 404 returned error can't find the container with id 0e90bd2953ec6dfa4104bc47e99ce28a4543898f12a087252eddae730da3afed
Sep 09 20:16:36.273496 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.273469    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"]
Sep 09 20:16:36.286609 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.286583    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-router-scheduler-695cbdhx9z"]
Sep 09 20:16:36.941989 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.941892    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" event={"ID":"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d","Type":"ContainerStarted","Data":"3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44"}
Sep 09 20:16:36.941989 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.941941    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" event={"ID":"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d","Type":"ContainerStarted","Data":"0e90bd2953ec6dfa4104bc47e99ce28a4543898f12a087252eddae730da3afed"}
Sep 09 20:16:36.942377 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.942021    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:16:36.988734 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:36.988677    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podStartSLOduration=1.9886606599999999 podStartE2EDuration="1.98866066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:36.986364661 +0000 UTC m=+1299.504502189" watchObservedRunningTime="2026-09-09 20:16:36.98866066 +0000 UTC m=+1299.506798187"
Sep 09 20:16:37.961850 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:37.961819    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" path="/var/lib/kubelet/pods/5dcb6ea4-7965-4235-ab90-15b2e1272d55/volumes"
Sep 09 20:16:56.392889 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.392852    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"]
Sep 09 20:16:56.393282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.393198    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main"
Sep 09 20:16:56.393282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.393210    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main"
Sep 09 20:16:56.393282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.393265    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dcb6ea4-7965-4235-ab90-15b2e1272d55" containerName="main"
Sep 09 20:16:56.396110 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.396087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.409518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.409491    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 09 20:16:56.409652 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.409535    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 09 20:16:56.409652 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.409493    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-jq6fb\""
Sep 09 20:16:56.409652 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.409501    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 09 20:16:56.409652 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.409493    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-qgtzg\""
Sep 09 20:16:56.427006 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.426973    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"]
Sep 09 20:16:56.450019 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.449978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cee996b-c7e5-42cc-8ddb-632cea4b2544-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.450203 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.450046    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9c5\" (UniqueName: \"kubernetes.io/projected/2cee996b-c7e5-42cc-8ddb-632cea4b2544-kube-api-access-dw9c5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.551400 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.551357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9c5\" (UniqueName: \"kubernetes.io/projected/2cee996b-c7e5-42cc-8ddb-632cea4b2544-kube-api-access-dw9c5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.551585 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.551429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cee996b-c7e5-42cc-8ddb-632cea4b2544-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.554057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.554031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cee996b-c7e5-42cc-8ddb-632cea4b2544-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.569876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.569845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9c5\" (UniqueName: \"kubernetes.io/projected/2cee996b-c7e5-42cc-8ddb-632cea4b2544-kube-api-access-dw9c5\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.705686 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.705591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:56.981367 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:56.981337    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"]
Sep 09 20:16:56.982756 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:16:56.982726    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cee996b_c7e5_42cc_8ddb_632cea4b2544.slice/crio-7a78d0a9849bf08f4dad637f5c4d7c87cb23c480ffcfdcd4d179502c07fdf8ca WatchSource:0}: Error finding container 7a78d0a9849bf08f4dad637f5c4d7c87cb23c480ffcfdcd4d179502c07fdf8ca: Status 404 returned error can't find the container with id 7a78d0a9849bf08f4dad637f5c4d7c87cb23c480ffcfdcd4d179502c07fdf8ca
Sep 09 20:16:57.041422 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:57.041392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" event={"ID":"2cee996b-c7e5-42cc-8ddb-632cea4b2544","Type":"ContainerStarted","Data":"7a78d0a9849bf08f4dad637f5c4d7c87cb23c480ffcfdcd4d179502c07fdf8ca"}
Sep 09 20:16:58.047353 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:58.047320    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" event={"ID":"2cee996b-c7e5-42cc-8ddb-632cea4b2544","Type":"ContainerStarted","Data":"25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd"}
Sep 09 20:16:58.047672 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:58.047389    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:16:58.069518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:16:58.069441    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" podStartSLOduration=2.069417461 podStartE2EDuration="2.069417461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:16:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:16:58.06735114 +0000 UTC m=+1320.585488668" watchObservedRunningTime="2026-09-09 20:16:58.069417461 +0000 UTC m=+1320.587554992"
Sep 09 20:17:07.948750 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:07.948721    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:17:29.054197 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:29.054123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:17:30.745987 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:30.745936    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"]
Sep 09 20:17:39.054568 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:39.054527    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:17:45.766068 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:45.766007    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main" containerID="cri-o://25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd" gracePeriod=45
Sep 09 20:17:49.054161 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:49.054126    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:17:59.054333 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:59.054282    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:17:59.054878 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:17:59.054383    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:18:09.054196 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:09.054153    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:18:16.014400 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.014376    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:18:16.097218 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.097192    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9c5\" (UniqueName: \"kubernetes.io/projected/2cee996b-c7e5-42cc-8ddb-632cea4b2544-kube-api-access-dw9c5\") pod \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") "
Sep 09 20:18:16.097371 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.097237    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cee996b-c7e5-42cc-8ddb-632cea4b2544-tls-certs\") pod \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\" (UID: \"2cee996b-c7e5-42cc-8ddb-632cea4b2544\") "
Sep 09 20:18:16.099608 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.099572    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cee996b-c7e5-42cc-8ddb-632cea4b2544-kube-api-access-dw9c5" (OuterVolumeSpecName: "kube-api-access-dw9c5") pod "2cee996b-c7e5-42cc-8ddb-632cea4b2544" (UID: "2cee996b-c7e5-42cc-8ddb-632cea4b2544"). InnerVolumeSpecName "kube-api-access-dw9c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:18:16.099608 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.099581    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cee996b-c7e5-42cc-8ddb-632cea4b2544-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2cee996b-c7e5-42cc-8ddb-632cea4b2544" (UID: "2cee996b-c7e5-42cc-8ddb-632cea4b2544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:18:16.198300 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.198266    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2cee996b-c7e5-42cc-8ddb-632cea4b2544-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.198300 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.198293    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9c5\" (UniqueName: \"kubernetes.io/projected/2cee996b-c7e5-42cc-8ddb-632cea4b2544-kube-api-access-dw9c5\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:18:16.423240 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.423148    2576 generic.go:356] "Generic (PLEG): container finished" podID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerID="25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd" exitCode=0
Sep 09 20:18:16.423240 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.423191    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"
Sep 09 20:18:16.423240 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.423203    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" event={"ID":"2cee996b-c7e5-42cc-8ddb-632cea4b2544","Type":"ContainerDied","Data":"25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd"}
Sep 09 20:18:16.423511 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.423256    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq" event={"ID":"2cee996b-c7e5-42cc-8ddb-632cea4b2544","Type":"ContainerDied","Data":"7a78d0a9849bf08f4dad637f5c4d7c87cb23c480ffcfdcd4d179502c07fdf8ca"}
Sep 09 20:18:16.423511 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.423285    2576 scope.go:117] "RemoveContainer" containerID="25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd"
Sep 09 20:18:16.432309 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.432292    2576 scope.go:117] "RemoveContainer" containerID="25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd"
Sep 09 20:18:16.432593 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:18:16.432575    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd\": container with ID starting with 25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd not found: ID does not exist" containerID="25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd"
Sep 09 20:18:16.432635 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.432603    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd"} err="failed to get container status \"25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd\": rpc error: code = NotFound desc = could not find container \"25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd\": container with ID starting with 25e3d2be76239b9d8881d3c133ba139540ef7de5447f5f41f450078918a9e3fd not found: ID does not exist"
Sep 09 20:18:16.446788 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.446761    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"]
Sep 09 20:18:16.451213 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:16.451181    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche7xscq"]
Sep 09 20:18:17.961790 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:17.961760    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" path="/var/lib/kubelet/pods/2cee996b-c7e5-42cc-8ddb-632cea4b2544/volumes"
Sep 09 20:18:33.271569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.271519    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"]
Sep 09 20:18:33.272146 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.272022    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main"
Sep 09 20:18:33.272146 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.272041    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main"
Sep 09 20:18:33.272146 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.272121    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cee996b-c7e5-42cc-8ddb-632cea4b2544" containerName="main"
Sep 09 20:18:33.274911 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.274891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.277414 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.277395    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 09 20:18:33.278403 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.278375    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 09 20:18:33.278518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.278401    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 09 20:18:33.278518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.278410    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-gbrpr\""
Sep 09 20:18:33.278518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.278481    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-wd594\""
Sep 09 20:18:33.289194 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.289170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"]
Sep 09 20:18:33.341148 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.341099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/2b31b9c7-3047-47cc-89ee-365b276157c9-kube-api-access-j8wbh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.341320 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.341191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b31b9c7-3047-47cc-89ee-365b276157c9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.441612 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.441567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/2b31b9c7-3047-47cc-89ee-365b276157c9-kube-api-access-j8wbh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.441778 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.441647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b31b9c7-3047-47cc-89ee-365b276157c9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.444520 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.444488    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b31b9c7-3047-47cc-89ee-365b276157c9-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.450203 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.450170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/2b31b9c7-3047-47cc-89ee-365b276157c9-kube-api-access-j8wbh\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.588875 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.588775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:33.863278 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:33.863243    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"]
Sep 09 20:18:33.866169 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:18:33.866137    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b31b9c7_3047_47cc_89ee_365b276157c9.slice/crio-d4e9570c2c1f523e2e730e59d10672dd4353b6bb5a7be73643875f377de4f790 WatchSource:0}: Error finding container d4e9570c2c1f523e2e730e59d10672dd4353b6bb5a7be73643875f377de4f790: Status 404 returned error can't find the container with id d4e9570c2c1f523e2e730e59d10672dd4353b6bb5a7be73643875f377de4f790
Sep 09 20:18:34.513107 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:34.513069    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" event={"ID":"2b31b9c7-3047-47cc-89ee-365b276157c9","Type":"ContainerStarted","Data":"82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1"}
Sep 09 20:18:34.513539 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:34.513126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" event={"ID":"2b31b9c7-3047-47cc-89ee-365b276157c9","Type":"ContainerStarted","Data":"d4e9570c2c1f523e2e730e59d10672dd4353b6bb5a7be73643875f377de4f790"}
Sep 09 20:18:34.513539 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:34.513179    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:18:34.532269 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:34.532214    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" podStartSLOduration=1.5321957739999998 podStartE2EDuration="1.532195774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:18:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:18:34.531779925 +0000 UTC m=+1417.049917454" watchObservedRunningTime="2026-09-09 20:18:34.532195774 +0000 UTC m=+1417.050333301"
Sep 09 20:18:46.970431 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:46.970395    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"]
Sep 09 20:18:47.948802 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:47.948763    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:18:57.949335 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:18:57.949296    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:02.000726 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:02.000657    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main" containerID="cri-o://3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44" gracePeriod=45
Sep 09 20:19:05.520256 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:05.520227    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:19:07.146948 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:07.146915    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"]
Sep 09 20:19:07.949129 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:07.949092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:07.949290 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:07.949195    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:19:15.519892 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:15.519855    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:17.948510 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:17.948471    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:22.168279 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:22.168223    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main" containerID="cri-o://82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1" gracePeriod=45
Sep 09 20:19:25.519668 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:25.519629    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:27.948752 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:27.948713    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:32.254107 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.254080    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:19:32.316279 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.316195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cldz\" (UniqueName: \"kubernetes.io/projected/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-kube-api-access-9cldz\") pod \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") "
Sep 09 20:19:32.316279 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.316237    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-tls-certs\") pod \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\" (UID: \"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d\") "
Sep 09 20:19:32.318537 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.318501    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-kube-api-access-9cldz" (OuterVolumeSpecName: "kube-api-access-9cldz") pod "aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" (UID: "aac95f3d-73e4-4e05-86f7-c0ec4f49c74d"). InnerVolumeSpecName "kube-api-access-9cldz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:32.318537 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.318519    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" (UID: "aac95f3d-73e4-4e05-86f7-c0ec4f49c74d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:32.416925 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.416896    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cldz\" (UniqueName: \"kubernetes.io/projected/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-kube-api-access-9cldz\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:19:32.416925 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.416921    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:19:32.804650 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.804613    2576 generic.go:356] "Generic (PLEG): container finished" podID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerID="3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44" exitCode=0
Sep 09 20:19:32.804650 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.804655    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"
Sep 09 20:19:32.804915 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.804676    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" event={"ID":"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d","Type":"ContainerDied","Data":"3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44"}
Sep 09 20:19:32.804915 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.804730    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg" event={"ID":"aac95f3d-73e4-4e05-86f7-c0ec4f49c74d","Type":"ContainerDied","Data":"0e90bd2953ec6dfa4104bc47e99ce28a4543898f12a087252eddae730da3afed"}
Sep 09 20:19:32.804915 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.804758    2576 scope.go:117] "RemoveContainer" containerID="3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44"
Sep 09 20:19:32.813504 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.813489    2576 scope.go:117] "RemoveContainer" containerID="3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44"
Sep 09 20:19:32.813749 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:19:32.813730    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44\": container with ID starting with 3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44 not found: ID does not exist" containerID="3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44"
Sep 09 20:19:32.813811 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.813757    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44"} err="failed to get container status \"3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44\": rpc error: code = NotFound desc = could not find container \"3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44\": container with ID starting with 3409cfbb1438163417f54867790384e1250ab61a8bd412107f1ee8aafa545f44 not found: ID does not exist"
Sep 09 20:19:32.826330 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.826308    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"]
Sep 09 20:19:32.832730 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:32.832711    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-invalid-token-rejected-e7cfa2c6/auth-invalid-token-test-kserve-router-scheduler-6dc5d5cf9cvxmtg"]
Sep 09 20:19:33.962401 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:33.962369    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" path="/var/lib/kubelet/pods/aac95f3d-73e4-4e05-86f7-c0ec4f49c74d/volumes"
Sep 09 20:19:35.519422 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:35.519383    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:35.519797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:35.519492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:19:42.688015 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.687983    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"]
Sep 09 20:19:42.688466 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.688303    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main"
Sep 09 20:19:42.688466 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.688314    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main"
Sep 09 20:19:42.688466 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.688373    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac95f3d-73e4-4e05-86f7-c0ec4f49c74d" containerName="main"
Sep 09 20:19:42.692754 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.692734    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:42.697367 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.697345    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 09 20:19:42.697562 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.697378    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 09 20:19:42.697689 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.697584    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-h9gqt\""
Sep 09 20:19:42.697689 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.697460    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 09 20:19:42.697689 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.697462    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-xx7c2\""
Sep 09 20:19:42.713535 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.713494    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"]
Sep 09 20:19:42.807149 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.807113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3a02ba-eed7-4702-9bca-ef9948278db7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:42.807330 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.807164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgdmf\" (UniqueName: \"kubernetes.io/projected/cf3a02ba-eed7-4702-9bca-ef9948278db7-kube-api-access-dgdmf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:42.908566 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.908523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3a02ba-eed7-4702-9bca-ef9948278db7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:42.908764 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.908587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgdmf\" (UniqueName: \"kubernetes.io/projected/cf3a02ba-eed7-4702-9bca-ef9948278db7-kube-api-access-dgdmf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:42.911136 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.911111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3a02ba-eed7-4702-9bca-ef9948278db7-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:42.917935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:42.917914    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgdmf\" (UniqueName: \"kubernetes.io/projected/cf3a02ba-eed7-4702-9bca-ef9948278db7-kube-api-access-dgdmf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:43.012303 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:43.012219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:43.297151 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:43.297119    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"]
Sep 09 20:19:43.298593 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:19:43.298568    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3a02ba_eed7_4702_9bca_ef9948278db7.slice/crio-f6f48e1891df1523dcdd2e3f5542030a67d3bd7d43ab842d8328776a26822eb3 WatchSource:0}: Error finding container f6f48e1891df1523dcdd2e3f5542030a67d3bd7d43ab842d8328776a26822eb3: Status 404 returned error can't find the container with id f6f48e1891df1523dcdd2e3f5542030a67d3bd7d43ab842d8328776a26822eb3
Sep 09 20:19:43.865660 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:43.865622    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" event={"ID":"cf3a02ba-eed7-4702-9bca-ef9948278db7","Type":"ContainerStarted","Data":"ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab"}
Sep 09 20:19:43.866131 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:43.865686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" event={"ID":"cf3a02ba-eed7-4702-9bca-ef9948278db7","Type":"ContainerStarted","Data":"f6f48e1891df1523dcdd2e3f5542030a67d3bd7d43ab842d8328776a26822eb3"}
Sep 09 20:19:43.866131 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:43.865799    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:19:43.888293 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:43.888240    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" podStartSLOduration=1.8882239429999998 podStartE2EDuration="1.888223943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:19:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:19:43.886202618 +0000 UTC m=+1486.404340135" watchObservedRunningTime="2026-09-09 20:19:43.888223943 +0000 UTC m=+1486.406361470"
Sep 09 20:19:45.520094 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:45.520057    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:19:52.421444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.421416    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:19:52.591649 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.591618    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/2b31b9c7-3047-47cc-89ee-365b276157c9-kube-api-access-j8wbh\") pod \"2b31b9c7-3047-47cc-89ee-365b276157c9\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") "
Sep 09 20:19:52.591817 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.591749    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b31b9c7-3047-47cc-89ee-365b276157c9-tls-certs\") pod \"2b31b9c7-3047-47cc-89ee-365b276157c9\" (UID: \"2b31b9c7-3047-47cc-89ee-365b276157c9\") "
Sep 09 20:19:52.593968 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.593929    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b31b9c7-3047-47cc-89ee-365b276157c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b31b9c7-3047-47cc-89ee-365b276157c9" (UID: "2b31b9c7-3047-47cc-89ee-365b276157c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:19:52.593968 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.593943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b31b9c7-3047-47cc-89ee-365b276157c9-kube-api-access-j8wbh" (OuterVolumeSpecName: "kube-api-access-j8wbh") pod "2b31b9c7-3047-47cc-89ee-365b276157c9" (UID: "2b31b9c7-3047-47cc-89ee-365b276157c9"). InnerVolumeSpecName "kube-api-access-j8wbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:19:52.692920 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.692885    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b31b9c7-3047-47cc-89ee-365b276157c9-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:19:52.692920 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.692913    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8wbh\" (UniqueName: \"kubernetes.io/projected/2b31b9c7-3047-47cc-89ee-365b276157c9-kube-api-access-j8wbh\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:19:52.909244 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.909146    2576 generic.go:356] "Generic (PLEG): container finished" podID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerID="82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1" exitCode=0
Sep 09 20:19:52.909244 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.909194    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"
Sep 09 20:19:52.909244 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.909234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" event={"ID":"2b31b9c7-3047-47cc-89ee-365b276157c9","Type":"ContainerDied","Data":"82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1"}
Sep 09 20:19:52.909521 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.909294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc" event={"ID":"2b31b9c7-3047-47cc-89ee-365b276157c9","Type":"ContainerDied","Data":"d4e9570c2c1f523e2e730e59d10672dd4353b6bb5a7be73643875f377de4f790"}
Sep 09 20:19:52.909521 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.909318    2576 scope.go:117] "RemoveContainer" containerID="82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1"
Sep 09 20:19:52.918409 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.918391    2576 scope.go:117] "RemoveContainer" containerID="82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1"
Sep 09 20:19:52.918736 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:19:52.918716    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1\": container with ID starting with 82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1 not found: ID does not exist" containerID="82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1"
Sep 09 20:19:52.918790 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.918746    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1"} err="failed to get container status \"82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1\": rpc error: code = NotFound desc = could not find container \"82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1\": container with ID starting with 82e933279858efb129ca419c5a71fae71569f6ef10e1f8023f5b37433a6d07f1 not found: ID does not exist"
Sep 09 20:19:52.931237 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.931213    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"]
Sep 09 20:19:52.935516 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:52.935496    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c98tscc"]
Sep 09 20:19:53.968189 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:53.968153    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" path="/var/lib/kubelet/pods/2b31b9c7-3047-47cc-89ee-365b276157c9/volumes"
Sep 09 20:19:58.000659 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:58.000628    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:19:58.001084 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:19:58.000637    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:20:14.873469 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:20:14.873440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:22:05.741704 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:05.741675    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"]
Sep 09 20:22:14.873675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:14.873632    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:20.762879 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:20.762831    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main" containerID="cri-o://ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab" gracePeriod=45
Sep 09 20:22:24.873024 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:24.872984    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:34.873504 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:34.873464    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:34.873893 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:34.873584    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:22:44.873480 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:44.873436    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:22:51.015533 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.015509    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:22:51.203557 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.203509    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgdmf\" (UniqueName: \"kubernetes.io/projected/cf3a02ba-eed7-4702-9bca-ef9948278db7-kube-api-access-dgdmf\") pod \"cf3a02ba-eed7-4702-9bca-ef9948278db7\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") "
Sep 09 20:22:51.203765 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.203580    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3a02ba-eed7-4702-9bca-ef9948278db7-tls-certs\") pod \"cf3a02ba-eed7-4702-9bca-ef9948278db7\" (UID: \"cf3a02ba-eed7-4702-9bca-ef9948278db7\") "
Sep 09 20:22:51.205841 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.205813    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf3a02ba-eed7-4702-9bca-ef9948278db7-kube-api-access-dgdmf" (OuterVolumeSpecName: "kube-api-access-dgdmf") pod "cf3a02ba-eed7-4702-9bca-ef9948278db7" (UID: "cf3a02ba-eed7-4702-9bca-ef9948278db7"). InnerVolumeSpecName "kube-api-access-dgdmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:22:51.205841 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.205822    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf3a02ba-eed7-4702-9bca-ef9948278db7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cf3a02ba-eed7-4702-9bca-ef9948278db7" (UID: "cf3a02ba-eed7-4702-9bca-ef9948278db7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:22:51.304108 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.304062    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgdmf\" (UniqueName: \"kubernetes.io/projected/cf3a02ba-eed7-4702-9bca-ef9948278db7-kube-api-access-dgdmf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:22:51.304108 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.304102    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf3a02ba-eed7-4702-9bca-ef9948278db7-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:22:51.764569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.764530    2576 generic.go:356] "Generic (PLEG): container finished" podID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerID="ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab" exitCode=0
Sep 09 20:22:51.764569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.764575    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"
Sep 09 20:22:51.764795 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.764623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" event={"ID":"cf3a02ba-eed7-4702-9bca-ef9948278db7","Type":"ContainerDied","Data":"ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab"}
Sep 09 20:22:51.764795 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.764675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4" event={"ID":"cf3a02ba-eed7-4702-9bca-ef9948278db7","Type":"ContainerDied","Data":"f6f48e1891df1523dcdd2e3f5542030a67d3bd7d43ab842d8328776a26822eb3"}
Sep 09 20:22:51.764795 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.764700    2576 scope.go:117] "RemoveContainer" containerID="ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab"
Sep 09 20:22:51.773473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.773456    2576 scope.go:117] "RemoveContainer" containerID="ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab"
Sep 09 20:22:51.773741 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:22:51.773723    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab\": container with ID starting with ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab not found: ID does not exist" containerID="ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab"
Sep 09 20:22:51.773808 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.773754    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab"} err="failed to get container status \"ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab\": rpc error: code = NotFound desc = could not find container \"ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab\": container with ID starting with ccfdaad1c60edfbdb961c28a2a3848a3b085483187c4f13b98b4eabca8b395ab not found: ID does not exist"
Sep 09 20:22:51.786101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.786073    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"]
Sep 09 20:22:51.790545 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.790522    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-qqhb4"]
Sep 09 20:22:51.961681 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:51.961652    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" path="/var/lib/kubelet/pods/cf3a02ba-eed7-4702-9bca-ef9948278db7/volumes"
Sep 09 20:22:54.849072 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849039    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"]
Sep 09 20:22:54.849429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849338    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main"
Sep 09 20:22:54.849429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849347    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main"
Sep 09 20:22:54.849429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849354    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main"
Sep 09 20:22:54.849429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849360    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main"
Sep 09 20:22:54.849429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849426    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf3a02ba-eed7-4702-9bca-ef9948278db7" containerName="main"
Sep 09 20:22:54.849615 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.849434    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b31b9c7-3047-47cc-89ee-365b276157c9" containerName="main"
Sep 09 20:22:54.853675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.853655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:54.856529 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.856503    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 09 20:22:54.857394 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.857349    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 09 20:22:54.857515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.857397    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-mgjrr\""
Sep 09 20:22:54.857515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.857408    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 09 20:22:54.868285 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.868263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"]
Sep 09 20:22:54.936603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.936564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:54.936603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.936621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e6f79-557c-4d79-b6a9-857fb7126aef-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:54.936875 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.936660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-dshm\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:54.936875 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.936709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-model-cache\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:54.936875 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.936735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-home\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:54.936875 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:54.936761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzl6\" (UniqueName: \"kubernetes.io/projected/f29e6f79-557c-4d79-b6a9-857fb7126aef-kube-api-access-hfzl6\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037383 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-model-cache\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037383 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-home\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037672 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzl6\" (UniqueName: \"kubernetes.io/projected/f29e6f79-557c-4d79-b6a9-857fb7126aef-kube-api-access-hfzl6\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037672 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037672 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e6f79-557c-4d79-b6a9-857fb7126aef-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037672 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-dshm\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037889 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-model-cache\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.037947 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.037923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.038068 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.038053    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-home\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.039968 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.039932    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-dshm\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.040204 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.040187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e6f79-557c-4d79-b6a9-857fb7126aef-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.046010 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.045986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzl6\" (UniqueName: \"kubernetes.io/projected/f29e6f79-557c-4d79-b6a9-857fb7126aef-kube-api-access-hfzl6\") pod \"canary-v1-kserve-65d58ccc9-9xgzj\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.164530 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.164440    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:22:55.441278 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.441251    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"]
Sep 09 20:22:55.443065 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:22:55.443028    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e6f79_557c_4d79_b6a9_857fb7126aef.slice/crio-77cf42bae2790c92b84c788433a1e4aaac5a574d7a48e8cdf8a560d5b9b07e1d WatchSource:0}: Error finding container 77cf42bae2790c92b84c788433a1e4aaac5a574d7a48e8cdf8a560d5b9b07e1d: Status 404 returned error can't find the container with id 77cf42bae2790c92b84c788433a1e4aaac5a574d7a48e8cdf8a560d5b9b07e1d
Sep 09 20:22:55.445031 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.445015    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:22:55.787155 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:55.787071    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj" event={"ID":"f29e6f79-557c-4d79-b6a9-857fb7126aef","Type":"ContainerStarted","Data":"77cf42bae2790c92b84c788433a1e4aaac5a574d7a48e8cdf8a560d5b9b07e1d"}
Sep 09 20:22:56.794071 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:56.794038    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj" event={"ID":"f29e6f79-557c-4d79-b6a9-857fb7126aef","Type":"ContainerStarted","Data":"5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a"}
Sep 09 20:22:56.814350 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:56.814257    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj" podStartSLOduration=1.695359327 podStartE2EDuration="2.814240938s" totalImagesPullingTime="1.118881611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:22:56.812213271 +0000 UTC m=+1679.330350799" watchObservedRunningTime="2026-09-09 20:22:56.814240938 +0000 UTC m=+1679.332378465"
Sep 09 20:22:57.164617 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:22:57.164578    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:23:05.165224 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:05.165182    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:23:05.178938 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:05.178909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:23:05.849182 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:05.849156    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:23:08.457043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.457008    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"]
Sep 09 20:23:08.460193 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.460174    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.463336 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.463317    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v2-kserve-self-signed-certs\""
Sep 09 20:23:08.478932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.478903    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"]
Sep 09 20:23:08.550917 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.550887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-dshm\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.551113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.550918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.551113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.550968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.551113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.551059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-home\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.551113 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.551101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znsq\" (UniqueName: \"kubernetes.io/projected/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-kube-api-access-9znsq\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.551282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.551134    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.651783 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.651745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-dshm\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.651783 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.651784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.651811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.651856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-home\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.651897    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znsq\" (UniqueName: \"kubernetes.io/projected/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-kube-api-access-9znsq\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652101 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.651930    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652357 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.652332    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-home\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652357 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.652349    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tmp-dir\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.652464 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.652390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-model-cache\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.654403 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.654371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-dshm\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.654608 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.654590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tls-certs\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.676612 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.676582    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znsq\" (UniqueName: \"kubernetes.io/projected/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-kube-api-access-9znsq\") pod \"canary-v2-kserve-7f65dbbccc-94dpd\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:08.770885 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:08.770792    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:09.060735 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:09.060710    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"]
Sep 09 20:23:09.062239 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:23:09.062216    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efe3055_f3ba_422a_8b82_c68cc0cab8fe.slice/crio-ed9048ef4510f79af076b3986281c59627488b338061942479841e7e3570ccb4 WatchSource:0}: Error finding container ed9048ef4510f79af076b3986281c59627488b338061942479841e7e3570ccb4: Status 404 returned error can't find the container with id ed9048ef4510f79af076b3986281c59627488b338061942479841e7e3570ccb4
Sep 09 20:23:09.860100 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:09.860064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" event={"ID":"6efe3055-f3ba-422a-8b82-c68cc0cab8fe","Type":"ContainerStarted","Data":"941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e"}
Sep 09 20:23:09.860559 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:09.860127    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" event={"ID":"6efe3055-f3ba-422a-8b82-c68cc0cab8fe","Type":"ContainerStarted","Data":"ed9048ef4510f79af076b3986281c59627488b338061942479841e7e3570ccb4"}
Sep 09 20:23:09.884347 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:09.884281    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" podStartSLOduration=1.884262039 podStartE2EDuration="1.884262039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:09.881000657 +0000 UTC m=+1692.399138188" watchObservedRunningTime="2026-09-09 20:23:09.884262039 +0000 UTC m=+1692.402399568"
Sep 09 20:23:10.771460 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:10.771417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:18.771238 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:18.771202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:18.784677 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:18.784651    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:18.915362 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:18.915289    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:23:34.214747 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.214716    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"]
Sep 09 20:23:34.217184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.217169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.220537 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.220510    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"kube-root-ca.crt\""
Sep 09 20:23:34.220675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.220557    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Sep 09 20:23:34.220675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.220631    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"openshift-service-ca.crt\""
Sep 09 20:23:34.221433 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.221417    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-fmscp\""
Sep 09 20:23:34.221500 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.221467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-9c386aa9\"/\"default-dockercfg-q2x8k\""
Sep 09 20:23:34.229158 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.229138    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"]
Sep 09 20:23:34.372331 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.372286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p58fz\" (UniqueName: \"kubernetes.io/projected/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-kube-api-access-p58fz\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.372500 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.372348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.473439 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.473344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p58fz\" (UniqueName: \"kubernetes.io/projected/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-kube-api-access-p58fz\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.473439 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.473420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.476128 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.476104    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.482705 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.482683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p58fz\" (UniqueName: \"kubernetes.io/projected/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-kube-api-access-p58fz\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") " pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.528629 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.528601    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:34.806451 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.806428    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"]
Sep 09 20:23:34.807985 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:23:34.807939    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d74e88_f8b7_4e4b_964a_fbe8a1acd3d0.slice/crio-e0b48bcb8e6eda46f7ff071ab6ddecf9b763e76919627f6caa8f07ceb8cfe59c WatchSource:0}: Error finding container e0b48bcb8e6eda46f7ff071ab6ddecf9b763e76919627f6caa8f07ceb8cfe59c: Status 404 returned error can't find the container with id e0b48bcb8e6eda46f7ff071ab6ddecf9b763e76919627f6caa8f07ceb8cfe59c
Sep 09 20:23:34.984096 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.984061    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" event={"ID":"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0","Type":"ContainerStarted","Data":"508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e"}
Sep 09 20:23:34.984316 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.984118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" event={"ID":"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0","Type":"ContainerStarted","Data":"e0b48bcb8e6eda46f7ff071ab6ddecf9b763e76919627f6caa8f07ceb8cfe59c"}
Sep 09 20:23:34.984316 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:34.984144    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:23:35.015436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:23:35.015383    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" podStartSLOduration=1.015368261 podStartE2EDuration="1.015368261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:23:35.0121097 +0000 UTC m=+1717.530247227" watchObservedRunningTime="2026-09-09 20:23:35.015368261 +0000 UTC m=+1717.533505840"
Sep 09 20:24:05.990757 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:24:05.990721    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:24:47.751375 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:24:47.751340    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"]
Sep 09 20:24:48.063403 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:24:48.063322    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"]
Sep 09 20:24:58.034099 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:24:58.034073    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:24:58.034546 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:24:58.034119    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:25:02.771848 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:02.771784    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" podUID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" containerName="main" containerID="cri-o://941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e" gracePeriod=45
Sep 09 20:25:02.905304 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:02.905268    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" podUID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 09 20:25:03.027321 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.027259    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:25:03.084774 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.084709    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj" podUID="f29e6f79-557c-4d79-b6a9-857fb7126aef" containerName="main" containerID="cri-o://5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a" gracePeriod=45
Sep 09 20:25:03.114184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114157    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tmp-dir\") pod \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") "
Sep 09 20:25:03.114313 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114221    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-model-cache\") pod \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") "
Sep 09 20:25:03.114313 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9znsq\" (UniqueName: \"kubernetes.io/projected/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-kube-api-access-9znsq\") pod \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") "
Sep 09 20:25:03.114313 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114275    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-dshm\") pod \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") "
Sep 09 20:25:03.114313 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114310    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-home\") pod \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") "
Sep 09 20:25:03.114515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114352    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tls-certs\") pod \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\" (UID: \"6efe3055-f3ba-422a-8b82-c68cc0cab8fe\") "
Sep 09 20:25:03.114515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114426    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6efe3055-f3ba-422a-8b82-c68cc0cab8fe" (UID: "6efe3055-f3ba-422a-8b82-c68cc0cab8fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.114515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114472    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "6efe3055-f3ba-422a-8b82-c68cc0cab8fe" (UID: "6efe3055-f3ba-422a-8b82-c68cc0cab8fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.114679 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114647    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.114679 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114647    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-home" (OuterVolumeSpecName: "home") pod "6efe3055-f3ba-422a-8b82-c68cc0cab8fe" (UID: "6efe3055-f3ba-422a-8b82-c68cc0cab8fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.114679 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.114666    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.116648 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.116616    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-dshm" (OuterVolumeSpecName: "dshm") pod "6efe3055-f3ba-422a-8b82-c68cc0cab8fe" (UID: "6efe3055-f3ba-422a-8b82-c68cc0cab8fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.116648 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.116627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6efe3055-f3ba-422a-8b82-c68cc0cab8fe" (UID: "6efe3055-f3ba-422a-8b82-c68cc0cab8fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:03.116648 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.116631    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-kube-api-access-9znsq" (OuterVolumeSpecName: "kube-api-access-9znsq") pod "6efe3055-f3ba-422a-8b82-c68cc0cab8fe" (UID: "6efe3055-f3ba-422a-8b82-c68cc0cab8fe"). InnerVolumeSpecName "kube-api-access-9znsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:03.215252 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.215222    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znsq\" (UniqueName: \"kubernetes.io/projected/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-kube-api-access-9znsq\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.215252 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.215250    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.215252 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.215259    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.215474 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.215267    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6efe3055-f3ba-422a-8b82-c68cc0cab8fe-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.325241 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.325217    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:25:03.416696 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.416658    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-home\") pod \"f29e6f79-557c-4d79-b6a9-857fb7126aef\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") "
Sep 09 20:25:03.416902 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.416735    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-dshm\") pod \"f29e6f79-557c-4d79-b6a9-857fb7126aef\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") "
Sep 09 20:25:03.416902 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.416759    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-tmp-dir\") pod \"f29e6f79-557c-4d79-b6a9-857fb7126aef\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") "
Sep 09 20:25:03.416902 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.416777    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e6f79-557c-4d79-b6a9-857fb7126aef-tls-certs\") pod \"f29e6f79-557c-4d79-b6a9-857fb7126aef\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") "
Sep 09 20:25:03.416902 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.416797    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-model-cache\") pod \"f29e6f79-557c-4d79-b6a9-857fb7126aef\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") "
Sep 09 20:25:03.416902 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.416826    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfzl6\" (UniqueName: \"kubernetes.io/projected/f29e6f79-557c-4d79-b6a9-857fb7126aef-kube-api-access-hfzl6\") pod \"f29e6f79-557c-4d79-b6a9-857fb7126aef\" (UID: \"f29e6f79-557c-4d79-b6a9-857fb7126aef\") "
Sep 09 20:25:03.417191 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.417082    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-home" (OuterVolumeSpecName: "home") pod "f29e6f79-557c-4d79-b6a9-857fb7126aef" (UID: "f29e6f79-557c-4d79-b6a9-857fb7126aef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.417191 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.417098    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f29e6f79-557c-4d79-b6a9-857fb7126aef" (UID: "f29e6f79-557c-4d79-b6a9-857fb7126aef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.417191 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.417178    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-model-cache" (OuterVolumeSpecName: "model-cache") pod "f29e6f79-557c-4d79-b6a9-857fb7126aef" (UID: "f29e6f79-557c-4d79-b6a9-857fb7126aef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.419233 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.419203    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-dshm" (OuterVolumeSpecName: "dshm") pod "f29e6f79-557c-4d79-b6a9-857fb7126aef" (UID: "f29e6f79-557c-4d79-b6a9-857fb7126aef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:03.419344 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.419231    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29e6f79-557c-4d79-b6a9-857fb7126aef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f29e6f79-557c-4d79-b6a9-857fb7126aef" (UID: "f29e6f79-557c-4d79-b6a9-857fb7126aef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:03.419344 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.419240    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29e6f79-557c-4d79-b6a9-857fb7126aef-kube-api-access-hfzl6" (OuterVolumeSpecName: "kube-api-access-hfzl6") pod "f29e6f79-557c-4d79-b6a9-857fb7126aef" (UID: "f29e6f79-557c-4d79-b6a9-857fb7126aef"). InnerVolumeSpecName "kube-api-access-hfzl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:03.425191 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.425161    2576 generic.go:356] "Generic (PLEG): container finished" podID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" containerID="941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e" exitCode=0
Sep 09 20:25:03.425335 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.425212    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"
Sep 09 20:25:03.425335 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.425245    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" event={"ID":"6efe3055-f3ba-422a-8b82-c68cc0cab8fe","Type":"ContainerDied","Data":"941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e"}
Sep 09 20:25:03.425335 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.425315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd" event={"ID":"6efe3055-f3ba-422a-8b82-c68cc0cab8fe","Type":"ContainerDied","Data":"ed9048ef4510f79af076b3986281c59627488b338061942479841e7e3570ccb4"}
Sep 09 20:25:03.425523 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.425346    2576 scope.go:117] "RemoveContainer" containerID="941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e"
Sep 09 20:25:03.426673 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.426649    2576 generic.go:356] "Generic (PLEG): container finished" podID="f29e6f79-557c-4d79-b6a9-857fb7126aef" containerID="5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a" exitCode=0
Sep 09 20:25:03.426779 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.426690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj" event={"ID":"f29e6f79-557c-4d79-b6a9-857fb7126aef","Type":"ContainerDied","Data":"5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a"}
Sep 09 20:25:03.426779 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.426727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj" event={"ID":"f29e6f79-557c-4d79-b6a9-857fb7126aef","Type":"ContainerDied","Data":"77cf42bae2790c92b84c788433a1e4aaac5a574d7a48e8cdf8a560d5b9b07e1d"}
Sep 09 20:25:03.426779 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.426695    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"
Sep 09 20:25:03.434988 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.434941    2576 scope.go:117] "RemoveContainer" containerID="941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e"
Sep 09 20:25:03.435294 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:25:03.435275    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e\": container with ID starting with 941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e not found: ID does not exist" containerID="941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e"
Sep 09 20:25:03.435354 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.435302    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e"} err="failed to get container status \"941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e\": rpc error: code = NotFound desc = could not find container \"941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e\": container with ID starting with 941492f5cc2d82c35b2fc079a23f586455b8ed765dc3ddf735f89927d5a7081e not found: ID does not exist"
Sep 09 20:25:03.435354 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.435318    2576 scope.go:117] "RemoveContainer" containerID="5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a"
Sep 09 20:25:03.443495 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.443479    2576 scope.go:117] "RemoveContainer" containerID="5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a"
Sep 09 20:25:03.443790 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:25:03.443768    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a\": container with ID starting with 5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a not found: ID does not exist" containerID="5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a"
Sep 09 20:25:03.443836 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.443797    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a"} err="failed to get container status \"5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a\": rpc error: code = NotFound desc = could not find container \"5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a\": container with ID starting with 5951c35235fa35af4047328a87dc2345b7b811e3f82b06e4a2b6b822f63deb0a not found: ID does not exist"
Sep 09 20:25:03.451819 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.451793    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"]
Sep 09 20:25:03.457824 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.457803    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v2-kserve-7f65dbbccc-94dpd"]
Sep 09 20:25:03.471212 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.471181    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"]
Sep 09 20:25:03.475556 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.475528    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-9xgzj"]
Sep 09 20:25:03.518132 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.518100    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.518132 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.518127    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f29e6f79-557c-4d79-b6a9-857fb7126aef-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.518132 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.518136    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.518327 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.518145    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzl6\" (UniqueName: \"kubernetes.io/projected/f29e6f79-557c-4d79-b6a9-857fb7126aef-kube-api-access-hfzl6\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.518327 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.518157    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.518327 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.518167    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f29e6f79-557c-4d79-b6a9-857fb7126aef-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:03.962947 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.962901    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" path="/var/lib/kubelet/pods/6efe3055-f3ba-422a-8b82-c68cc0cab8fe/volumes"
Sep 09 20:25:03.963487 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:03.963465    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29e6f79-557c-4d79-b6a9-857fb7126aef" path="/var/lib/kubelet/pods/f29e6f79-557c-4d79-b6a9-857fb7126aef/volumes"
Sep 09 20:25:04.646176 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646139    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"]
Sep 09 20:25:04.646481 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646469    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29e6f79-557c-4d79-b6a9-857fb7126aef" containerName="main"
Sep 09 20:25:04.646526 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646482    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29e6f79-557c-4d79-b6a9-857fb7126aef" containerName="main"
Sep 09 20:25:04.646526 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646496    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" containerName="main"
Sep 09 20:25:04.646526 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646501    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" containerName="main"
Sep 09 20:25:04.646621 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646567    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29e6f79-557c-4d79-b6a9-857fb7126aef" containerName="main"
Sep 09 20:25:04.646621 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.646576    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6efe3055-f3ba-422a-8b82-c68cc0cab8fe" containerName="main"
Sep 09 20:25:04.649296 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.649280    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.651905 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.651886    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 09 20:25:04.652114 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.652096    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 09 20:25:04.652345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.652330    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-454ck\""
Sep 09 20:25:04.652846 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.652831    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 09 20:25:04.660830 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.660809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"]
Sep 09 20:25:04.728194 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.728160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-home\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.728374 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.728239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/1a8e840c-06da-40b1-b491-702a616e35f1-kube-api-access-vgkxf\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.728374 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.728266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.728374 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.728350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8e840c-06da-40b1-b491-702a616e35f1-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.728496 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.728392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.728496 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.728421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.829556 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.829502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.829556 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.829565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.829839 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.829592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-home\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.829839 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.829639    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/1a8e840c-06da-40b1-b491-702a616e35f1-kube-api-access-vgkxf\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.829839 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.829662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.829839 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.829708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8e840c-06da-40b1-b491-702a616e35f1-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.830126 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.830094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-home\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.830199 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.830127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.830199 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.830154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.832164 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.832140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.832438 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.832420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8e840c-06da-40b1-b491-702a616e35f1-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.841192 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.841167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/1a8e840c-06da-40b1-b491-702a616e35f1-kube-api-access-vgkxf\") pod \"test-div-v2-kserve-5f6fdb6b77-5bcdd\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:04.959843 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:04.959744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:05.239489 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:05.239461    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"]
Sep 09 20:25:05.240675 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:25:05.240644    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8e840c_06da_40b1_b491_702a616e35f1.slice/crio-da108f2e8fa27c0afc654758daeaedf2c762c5f02d6a5780f0002d7470465121 WatchSource:0}: Error finding container da108f2e8fa27c0afc654758daeaedf2c762c5f02d6a5780f0002d7470465121: Status 404 returned error can't find the container with id da108f2e8fa27c0afc654758daeaedf2c762c5f02d6a5780f0002d7470465121
Sep 09 20:25:05.445231 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:05.445191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd" event={"ID":"1a8e840c-06da-40b1-b491-702a616e35f1","Type":"ContainerStarted","Data":"b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318"}
Sep 09 20:25:05.445380 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:05.445253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd" event={"ID":"1a8e840c-06da-40b1-b491-702a616e35f1","Type":"ContainerStarted","Data":"da108f2e8fa27c0afc654758daeaedf2c762c5f02d6a5780f0002d7470465121"}
Sep 09 20:25:05.962006 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:05.961974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:14.960256 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:14.960219    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:14.973215 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:14.973189    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:14.994490 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:14.994437    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd" podStartSLOduration=10.994420616 podStartE2EDuration="10.994420616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:25:05.468146464 +0000 UTC m=+1807.986283992" watchObservedRunningTime="2026-09-09 20:25:14.994420616 +0000 UTC m=+1817.512558142"
Sep 09 20:25:15.507064 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:15.507038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:20.412882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:20.412851    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"]
Sep 09 20:25:23.590861 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:23.590825    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"]
Sep 09 20:25:25.990336 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:25.990292    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:25:35.434235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:35.434173    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main" containerID="cri-o://508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e" gracePeriod=45
Sep 09 20:25:35.990881 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:35.990838    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:25:38.611474 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.611432    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd" podUID="1a8e840c-06da-40b1-b491-702a616e35f1" containerName="main" containerID="cri-o://b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318" gracePeriod=45
Sep 09 20:25:38.854379 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.854354    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:38.919554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919465    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/1a8e840c-06da-40b1-b491-702a616e35f1-kube-api-access-vgkxf\") pod \"1a8e840c-06da-40b1-b491-702a616e35f1\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") "
Sep 09 20:25:38.919554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8e840c-06da-40b1-b491-702a616e35f1-tls-certs\") pod \"1a8e840c-06da-40b1-b491-702a616e35f1\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") "
Sep 09 20:25:38.919554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919520    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-model-cache\") pod \"1a8e840c-06da-40b1-b491-702a616e35f1\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") "
Sep 09 20:25:38.919554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919551    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-tmp-dir\") pod \"1a8e840c-06da-40b1-b491-702a616e35f1\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") "
Sep 09 20:25:38.919880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919603    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-home\") pod \"1a8e840c-06da-40b1-b491-702a616e35f1\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") "
Sep 09 20:25:38.919880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919632    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-dshm\") pod \"1a8e840c-06da-40b1-b491-702a616e35f1\" (UID: \"1a8e840c-06da-40b1-b491-702a616e35f1\") "
Sep 09 20:25:38.919880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.919844    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a8e840c-06da-40b1-b491-702a616e35f1" (UID: "1a8e840c-06da-40b1-b491-702a616e35f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.920426 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.920358    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-home" (OuterVolumeSpecName: "home") pod "1a8e840c-06da-40b1-b491-702a616e35f1" (UID: "1a8e840c-06da-40b1-b491-702a616e35f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.920563 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.920437    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a8e840c-06da-40b1-b491-702a616e35f1" (UID: "1a8e840c-06da-40b1-b491-702a616e35f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.922915 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.922880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8e840c-06da-40b1-b491-702a616e35f1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a8e840c-06da-40b1-b491-702a616e35f1" (UID: "1a8e840c-06da-40b1-b491-702a616e35f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:25:38.922915 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.922902    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-dshm" (OuterVolumeSpecName: "dshm") pod "1a8e840c-06da-40b1-b491-702a616e35f1" (UID: "1a8e840c-06da-40b1-b491-702a616e35f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:25:38.927759 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:38.927722    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8e840c-06da-40b1-b491-702a616e35f1-kube-api-access-vgkxf" (OuterVolumeSpecName: "kube-api-access-vgkxf") pod "1a8e840c-06da-40b1-b491-702a616e35f1" (UID: "1a8e840c-06da-40b1-b491-702a616e35f1"). InnerVolumeSpecName "kube-api-access-vgkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:25:39.021069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.021036    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgkxf\" (UniqueName: \"kubernetes.io/projected/1a8e840c-06da-40b1-b491-702a616e35f1-kube-api-access-vgkxf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.021069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.021062    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a8e840c-06da-40b1-b491-702a616e35f1-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.021069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.021072    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.021069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.021080    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.021325 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.021088    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.021325 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.021095    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a8e840c-06da-40b1-b491-702a616e35f1-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:25:39.618843 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.618807    2576 generic.go:356] "Generic (PLEG): container finished" podID="1a8e840c-06da-40b1-b491-702a616e35f1" containerID="b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318" exitCode=0
Sep 09 20:25:39.619332 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.618870    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"
Sep 09 20:25:39.619332 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.618893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd" event={"ID":"1a8e840c-06da-40b1-b491-702a616e35f1","Type":"ContainerDied","Data":"b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318"}
Sep 09 20:25:39.619332 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.618946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd" event={"ID":"1a8e840c-06da-40b1-b491-702a616e35f1","Type":"ContainerDied","Data":"da108f2e8fa27c0afc654758daeaedf2c762c5f02d6a5780f0002d7470465121"}
Sep 09 20:25:39.619332 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.618988    2576 scope.go:117] "RemoveContainer" containerID="b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318"
Sep 09 20:25:39.628374 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.628359    2576 scope.go:117] "RemoveContainer" containerID="b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318"
Sep 09 20:25:39.628625 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:25:39.628607    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318\": container with ID starting with b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318 not found: ID does not exist" containerID="b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318"
Sep 09 20:25:39.628675 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.628630    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318"} err="failed to get container status \"b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318\": rpc error: code = NotFound desc = could not find container \"b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318\": container with ID starting with b17afbbff52e1e5d5e47340124ad0e3fe16b4c935a5548e446221d3b96bdf318 not found: ID does not exist"
Sep 09 20:25:39.653293 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.653272    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"]
Sep 09 20:25:39.663086 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.663059    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v2-kserve-5f6fdb6b77-5bcdd"]
Sep 09 20:25:39.962153 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:39.962065    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a8e840c-06da-40b1-b491-702a616e35f1" path="/var/lib/kubelet/pods/1a8e840c-06da-40b1-b491-702a616e35f1/volumes"
Sep 09 20:25:45.990412 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:45.990378    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:25:45.990882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:45.990507    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:25:55.991123 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:25:55.991084    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:26:05.690941 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.690879    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:26:05.740820 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.740791    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-tls-certs\") pod \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") "
Sep 09 20:26:05.741029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.740866    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p58fz\" (UniqueName: \"kubernetes.io/projected/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-kube-api-access-p58fz\") pod \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\" (UID: \"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0\") "
Sep 09 20:26:05.743270 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.743239    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" (UID: "31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:26:05.743270 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.743251    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-kube-api-access-p58fz" (OuterVolumeSpecName: "kube-api-access-p58fz") pod "31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" (UID: "31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0"). InnerVolumeSpecName "kube-api-access-p58fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:26:05.746261 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.746230    2576 generic.go:356] "Generic (PLEG): container finished" podID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerID="508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e" exitCode=0
Sep 09 20:26:05.746361 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.746275    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"
Sep 09 20:26:05.746361 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.746313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" event={"ID":"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0","Type":"ContainerDied","Data":"508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e"}
Sep 09 20:26:05.746440 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.746387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v" event={"ID":"31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0","Type":"ContainerDied","Data":"e0b48bcb8e6eda46f7ff071ab6ddecf9b763e76919627f6caa8f07ceb8cfe59c"}
Sep 09 20:26:05.746440 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.746409    2576 scope.go:117] "RemoveContainer" containerID="508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e"
Sep 09 20:26:05.759904 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.759886    2576 scope.go:117] "RemoveContainer" containerID="508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e"
Sep 09 20:26:05.760204 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:26:05.760186    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e\": container with ID starting with 508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e not found: ID does not exist" containerID="508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e"
Sep 09 20:26:05.760245 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.760214    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e"} err="failed to get container status \"508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e\": rpc error: code = NotFound desc = could not find container \"508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e\": container with ID starting with 508fc090a90e4a923a3d2b1b171636e1755926ecc4b196ad2b278bde6baa859e not found: ID does not exist"
Sep 09 20:26:05.770882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.770857    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"]
Sep 09 20:26:05.774599 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.774579    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-9c386aa9/scheduler-v06-pd-migration-test-kserve-router-scheduler-75b2q8v"]
Sep 09 20:26:05.841625 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.841595    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p58fz\" (UniqueName: \"kubernetes.io/projected/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-kube-api-access-p58fz\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:26:05.841625 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.841622    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:26:05.961562 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:05.961475    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" path="/var/lib/kubelet/pods/31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0/volumes"
Sep 09 20:26:13.946315 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946280    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"]
Sep 09 20:26:13.946684 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946633    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a8e840c-06da-40b1-b491-702a616e35f1" containerName="main"
Sep 09 20:26:13.946684 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946644    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a8e840c-06da-40b1-b491-702a616e35f1" containerName="main"
Sep 09 20:26:13.946684 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946668    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main"
Sep 09 20:26:13.946684 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946674    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main"
Sep 09 20:26:13.946825 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946721    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="31d74e88-f8b7-4e4b-964a-fbe8a1acd3d0" containerName="main"
Sep 09 20:26:13.946825 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.946733    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a8e840c-06da-40b1-b491-702a616e35f1" containerName="main"
Sep 09 20:26:13.951183 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.951162    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:13.955197 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.955169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v1-kserve-self-signed-certs\""
Sep 09 20:26:13.955197 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.955183    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 09 20:26:13.955391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.955184    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-zmzpq\""
Sep 09 20:26:13.955391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.955268    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 09 20:26:13.963605 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:13.963579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"]
Sep 09 20:26:14.111635 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.111593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-home\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.111635 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.111643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgcnq\" (UniqueName: \"kubernetes.io/projected/cba2d642-05ff-4d04-98e9-4430766adc35-kube-api-access-bgcnq\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.111880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.111664    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-dshm\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.111880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.111680    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba2d642-05ff-4d04-98e9-4430766adc35-tls-certs\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.111880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.111742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-model-cache\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.111880 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.111800    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213207 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgcnq\" (UniqueName: \"kubernetes.io/projected/cba2d642-05ff-4d04-98e9-4430766adc35-kube-api-access-bgcnq\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213207 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-dshm\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213207 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba2d642-05ff-4d04-98e9-4430766adc35-tls-certs\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213462 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213293    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-model-cache\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213462 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213330    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213579 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-home\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213652 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-model-cache\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.213766 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-tmp-dir\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.214088 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.213785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-home\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.215583 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.215561    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-dshm\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.215826 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.215811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba2d642-05ff-4d04-98e9-4430766adc35-tls-certs\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.223140 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.223121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgcnq\" (UniqueName: \"kubernetes.io/projected/cba2d642-05ff-4d04-98e9-4430766adc35-kube-api-access-bgcnq\") pod \"stop-v1-kserve-5657b88c95-mblml\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") " pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.261139 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.261107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:14.541934 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.541902    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"]
Sep 09 20:26:14.543028 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:26:14.543004    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcba2d642_05ff_4d04_98e9_4430766adc35.slice/crio-9e2abae14b1b45e877f97ff60f80cacb5380716dd1e46673216683491174fe5f WatchSource:0}: Error finding container 9e2abae14b1b45e877f97ff60f80cacb5380716dd1e46673216683491174fe5f: Status 404 returned error can't find the container with id 9e2abae14b1b45e877f97ff60f80cacb5380716dd1e46673216683491174fe5f
Sep 09 20:26:14.793000 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.792901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" event={"ID":"cba2d642-05ff-4d04-98e9-4430766adc35","Type":"ContainerStarted","Data":"1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d"}
Sep 09 20:26:14.793000 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.792968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" event={"ID":"cba2d642-05ff-4d04-98e9-4430766adc35","Type":"ContainerStarted","Data":"9e2abae14b1b45e877f97ff60f80cacb5380716dd1e46673216683491174fe5f"}
Sep 09 20:26:14.810973 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:14.810898    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" podStartSLOduration=1.810882051 podStartE2EDuration="1.810882051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:14.809811888 +0000 UTC m=+1877.327949415" watchObservedRunningTime="2026-09-09 20:26:14.810882051 +0000 UTC m=+1877.329019623"
Sep 09 20:26:15.261239 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:15.261192    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:24.261995 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:24.261873    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:24.275034 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:24.275002    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:24.853440 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:24.853405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:26:31.629612 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.629579    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"]
Sep 09 20:26:31.633232 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.633211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.635518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.635498    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 09 20:26:31.643115 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.643092    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"]
Sep 09 20:26:31.762765 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.762727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tls-certs\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.762765 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.762767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-dshm\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.762985 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.762790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-home\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.762985 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.762826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-model-cache\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.762985 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.762847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.762985 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.762911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhmkf\" (UniqueName: \"kubernetes.io/projected/1592c5a2-f5b3-479e-82b0-2cbee43d5888-kube-api-access-dhmkf\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864021 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.863982    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhmkf\" (UniqueName: \"kubernetes.io/projected/1592c5a2-f5b3-479e-82b0-2cbee43d5888-kube-api-access-dhmkf\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864225 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tls-certs\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864225 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864082    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-dshm\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864225 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-home\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864225 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864131    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-model-cache\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864225 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864585 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-model-cache\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864662 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864565    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-home\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.864662 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.864654    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.867108 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.867080    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-dshm\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.867424 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.867400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tls-certs\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.872841 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.872812    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhmkf\" (UniqueName: \"kubernetes.io/projected/1592c5a2-f5b3-479e-82b0-2cbee43d5888-kube-api-access-dhmkf\") pod \"stop-v2-kserve-7f68747c45-gtb22\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:31.944568 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:31.944471    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:32.222392 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:32.222362    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"]
Sep 09 20:26:32.223768 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:26:32.223740    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1592c5a2_f5b3_479e_82b0_2cbee43d5888.slice/crio-40f70fa4f6d791d9e912089e479abbce7624fd874f23fbc3b5109fab18edbc07 WatchSource:0}: Error finding container 40f70fa4f6d791d9e912089e479abbce7624fd874f23fbc3b5109fab18edbc07: Status 404 returned error can't find the container with id 40f70fa4f6d791d9e912089e479abbce7624fd874f23fbc3b5109fab18edbc07
Sep 09 20:26:32.884673 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:32.884631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22" event={"ID":"1592c5a2-f5b3-479e-82b0-2cbee43d5888","Type":"ContainerStarted","Data":"354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf"}
Sep 09 20:26:32.885126 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:32.884686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22" event={"ID":"1592c5a2-f5b3-479e-82b0-2cbee43d5888","Type":"ContainerStarted","Data":"40f70fa4f6d791d9e912089e479abbce7624fd874f23fbc3b5109fab18edbc07"}
Sep 09 20:26:32.902463 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:32.902416    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22" podStartSLOduration=1.902402742 podStartE2EDuration="1.902402742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:32.899647909 +0000 UTC m=+1895.417785435" watchObservedRunningTime="2026-09-09 20:26:32.902402742 +0000 UTC m=+1895.420540274"
Sep 09 20:26:32.945043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:32.945011    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:41.945403 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:41.945366    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:41.961609 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:41.961582    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:42.946765 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:42.946731    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:26:43.785772 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.785735    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"]
Sep 09 20:26:43.789335 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.789310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:43.792815 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.792788    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-22988\""
Sep 09 20:26:43.792815 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.792788    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 09 20:26:43.793029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.792841    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 09 20:26:43.793029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.792850    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-epp-sa-dockercfg-j2kkq\""
Sep 09 20:26:43.793029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.792913    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 09 20:26:43.800250 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.800226    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"]
Sep 09 20:26:43.862198 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.862167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4fx\" (UniqueName: \"kubernetes.io/projected/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-kube-api-access-mh4fx\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:43.862357 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.862268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:43.963656 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.963627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:43.964129 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.963682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4fx\" (UniqueName: \"kubernetes.io/projected/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-kube-api-access-mh4fx\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:43.966762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.966736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:43.972366 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:43.972338    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4fx\" (UniqueName: \"kubernetes.io/projected/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-kube-api-access-mh4fx\") pod \"scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:44.103539 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:44.103501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:44.375475 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:44.375445    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"]
Sep 09 20:26:44.377440 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:26:44.377410    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3487a2ff_027c_4490_9ab2_5b6154c5e6d8.slice/crio-86cdb22d5001264d567c3b90ff30c9756a4a4be599fae33e8a772ed0f432c7e8 WatchSource:0}: Error finding container 86cdb22d5001264d567c3b90ff30c9756a4a4be599fae33e8a772ed0f432c7e8: Status 404 returned error can't find the container with id 86cdb22d5001264d567c3b90ff30c9756a4a4be599fae33e8a772ed0f432c7e8
Sep 09 20:26:44.947385 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:44.947348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" event={"ID":"3487a2ff-027c-4490-9ab2-5b6154c5e6d8","Type":"ContainerStarted","Data":"60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4"}
Sep 09 20:26:44.947597 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:44.947413    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:26:44.947597 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:44.947437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" event={"ID":"3487a2ff-027c-4490-9ab2-5b6154c5e6d8","Type":"ContainerStarted","Data":"86cdb22d5001264d567c3b90ff30c9756a4a4be599fae33e8a772ed0f432c7e8"}
Sep 09 20:26:44.966849 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:26:44.966782    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" podStartSLOduration=1.966763019 podStartE2EDuration="1.966763019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:26:44.965266895 +0000 UTC m=+1907.483404422" watchObservedRunningTime="2026-09-09 20:26:44.966763019 +0000 UTC m=+1907.484900562"
Sep 09 20:27:15.953449 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:15.953419    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:27:20.365355 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:20.365319    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"]
Sep 09 20:27:25.102384 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:25.102348    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"]
Sep 09 20:27:25.953282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:25.953237    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:27:35.385649 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:35.385607    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main" containerID="cri-o://60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4" gracePeriod=45
Sep 09 20:27:35.953645 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:35.953606    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:27:40.125429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.125370    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22" podUID="1592c5a2-f5b3-479e-82b0-2cbee43d5888" containerName="main" containerID="cri-o://354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf" gracePeriod=45
Sep 09 20:27:40.373073 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.373049    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:27:40.459461 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459366    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tls-certs\") pod \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") "
Sep 09 20:27:40.459461 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459422    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-home\") pod \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") "
Sep 09 20:27:40.459694 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459479    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tmp-dir\") pod \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") "
Sep 09 20:27:40.459694 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459505    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhmkf\" (UniqueName: \"kubernetes.io/projected/1592c5a2-f5b3-479e-82b0-2cbee43d5888-kube-api-access-dhmkf\") pod \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") "
Sep 09 20:27:40.459694 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459552    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-dshm\") pod \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") "
Sep 09 20:27:40.459694 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459599    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-model-cache\") pod \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\" (UID: \"1592c5a2-f5b3-479e-82b0-2cbee43d5888\") "
Sep 09 20:27:40.459903 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459772    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-home" (OuterVolumeSpecName: "home") pod "1592c5a2-f5b3-479e-82b0-2cbee43d5888" (UID: "1592c5a2-f5b3-479e-82b0-2cbee43d5888"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:40.459903 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459781    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1592c5a2-f5b3-479e-82b0-2cbee43d5888" (UID: "1592c5a2-f5b3-479e-82b0-2cbee43d5888"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:40.459903 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459889    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-model-cache" (OuterVolumeSpecName: "model-cache") pod "1592c5a2-f5b3-479e-82b0-2cbee43d5888" (UID: "1592c5a2-f5b3-479e-82b0-2cbee43d5888"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:40.460058 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459944    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:27:40.460058 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.459994    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:27:40.461891 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.461863    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1592c5a2-f5b3-479e-82b0-2cbee43d5888" (UID: "1592c5a2-f5b3-479e-82b0-2cbee43d5888"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:27:40.462014 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.461911    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1592c5a2-f5b3-479e-82b0-2cbee43d5888-kube-api-access-dhmkf" (OuterVolumeSpecName: "kube-api-access-dhmkf") pod "1592c5a2-f5b3-479e-82b0-2cbee43d5888" (UID: "1592c5a2-f5b3-479e-82b0-2cbee43d5888"). InnerVolumeSpecName "kube-api-access-dhmkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:27:40.462014 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.461914    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-dshm" (OuterVolumeSpecName: "dshm") pod "1592c5a2-f5b3-479e-82b0-2cbee43d5888" (UID: "1592c5a2-f5b3-479e-82b0-2cbee43d5888"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:27:40.561289 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.561251    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhmkf\" (UniqueName: \"kubernetes.io/projected/1592c5a2-f5b3-479e-82b0-2cbee43d5888-kube-api-access-dhmkf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:27:40.561289 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.561283    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:27:40.561289 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.561294    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1592c5a2-f5b3-479e-82b0-2cbee43d5888-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:27:40.561533 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:40.561303    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1592c5a2-f5b3-479e-82b0-2cbee43d5888-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:27:41.225539 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.225492    2576 generic.go:356] "Generic (PLEG): container finished" podID="1592c5a2-f5b3-479e-82b0-2cbee43d5888" containerID="354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf" exitCode=0
Sep 09 20:27:41.225991 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.225546    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"
Sep 09 20:27:41.225991 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.225574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22" event={"ID":"1592c5a2-f5b3-479e-82b0-2cbee43d5888","Type":"ContainerDied","Data":"354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf"}
Sep 09 20:27:41.225991 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.225632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22" event={"ID":"1592c5a2-f5b3-479e-82b0-2cbee43d5888","Type":"ContainerDied","Data":"40f70fa4f6d791d9e912089e479abbce7624fd874f23fbc3b5109fab18edbc07"}
Sep 09 20:27:41.225991 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.225653    2576 scope.go:117] "RemoveContainer" containerID="354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf"
Sep 09 20:27:41.234762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.234746    2576 scope.go:117] "RemoveContainer" containerID="354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf"
Sep 09 20:27:41.235072 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:27:41.235052    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf\": container with ID starting with 354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf not found: ID does not exist" containerID="354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf"
Sep 09 20:27:41.235117 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.235083    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf"} err="failed to get container status \"354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf\": rpc error: code = NotFound desc = could not find container \"354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf\": container with ID starting with 354ba8452c242ca29cd0d2a2b3427989f25b559a1272275a9c9ad0ba937459bf not found: ID does not exist"
Sep 09 20:27:41.246853 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.246832    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"]
Sep 09 20:27:41.250226 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.250205    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-gtb22"]
Sep 09 20:27:41.962213 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:41.962172    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1592c5a2-f5b3-479e-82b0-2cbee43d5888" path="/var/lib/kubelet/pods/1592c5a2-f5b3-479e-82b0-2cbee43d5888/volumes"
Sep 09 20:27:45.953469 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:45.953427    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:27:45.953878 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:45.953539    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:27:50.659366 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:50.659329    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"]
Sep 09 20:27:55.953594 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:27:55.953551    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:28:05.631503 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.631478    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:28:05.682830 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.682732    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" podUID="cba2d642-05ff-4d04-98e9-4430766adc35" containerName="main" containerID="cri-o://1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d" gracePeriod=45
Sep 09 20:28:05.784111 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.784078    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-tls-certs\") pod \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") "
Sep 09 20:28:05.784279 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.784137    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh4fx\" (UniqueName: \"kubernetes.io/projected/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-kube-api-access-mh4fx\") pod \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\" (UID: \"3487a2ff-027c-4490-9ab2-5b6154c5e6d8\") "
Sep 09 20:28:05.786561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.786513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3487a2ff-027c-4490-9ab2-5b6154c5e6d8" (UID: "3487a2ff-027c-4490-9ab2-5b6154c5e6d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:05.786561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.786535    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-kube-api-access-mh4fx" (OuterVolumeSpecName: "kube-api-access-mh4fx") pod "3487a2ff-027c-4490-9ab2-5b6154c5e6d8" (UID: "3487a2ff-027c-4490-9ab2-5b6154c5e6d8"). InnerVolumeSpecName "kube-api-access-mh4fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:05.843316 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.843264    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" podUID="cba2d642-05ff-4d04-98e9-4430766adc35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/health\": dial tcp 10.133.0.50:8000: connect: connection refused"
Sep 09 20:28:05.885125 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.885097    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh4fx\" (UniqueName: \"kubernetes.io/projected/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-kube-api-access-mh4fx\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:05.885125 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.885124    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3487a2ff-027c-4490-9ab2-5b6154c5e6d8-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:05.931387 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:05.931362    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:28:06.086843 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.086809    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-home\") pod \"cba2d642-05ff-4d04-98e9-4430766adc35\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") "
Sep 09 20:28:06.086843 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.086844    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-tmp-dir\") pod \"cba2d642-05ff-4d04-98e9-4430766adc35\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") "
Sep 09 20:28:06.087138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.086866    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgcnq\" (UniqueName: \"kubernetes.io/projected/cba2d642-05ff-4d04-98e9-4430766adc35-kube-api-access-bgcnq\") pod \"cba2d642-05ff-4d04-98e9-4430766adc35\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") "
Sep 09 20:28:06.087138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.086892    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-model-cache\") pod \"cba2d642-05ff-4d04-98e9-4430766adc35\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") "
Sep 09 20:28:06.087138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.086945    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-dshm\") pod \"cba2d642-05ff-4d04-98e9-4430766adc35\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") "
Sep 09 20:28:06.087138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.086992    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba2d642-05ff-4d04-98e9-4430766adc35-tls-certs\") pod \"cba2d642-05ff-4d04-98e9-4430766adc35\" (UID: \"cba2d642-05ff-4d04-98e9-4430766adc35\") "
Sep 09 20:28:06.087138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.087124    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-home" (OuterVolumeSpecName: "home") pod "cba2d642-05ff-4d04-98e9-4430766adc35" (UID: "cba2d642-05ff-4d04-98e9-4430766adc35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:06.087391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.087152    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cba2d642-05ff-4d04-98e9-4430766adc35" (UID: "cba2d642-05ff-4d04-98e9-4430766adc35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:06.087391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.087197    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-model-cache" (OuterVolumeSpecName: "model-cache") pod "cba2d642-05ff-4d04-98e9-4430766adc35" (UID: "cba2d642-05ff-4d04-98e9-4430766adc35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:06.087391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.087292    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:06.087391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.087305    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:06.087391 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.087315    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:06.089282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.089260    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-dshm" (OuterVolumeSpecName: "dshm") pod "cba2d642-05ff-4d04-98e9-4430766adc35" (UID: "cba2d642-05ff-4d04-98e9-4430766adc35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:28:06.089282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.089258    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cba2d642-05ff-4d04-98e9-4430766adc35-kube-api-access-bgcnq" (OuterVolumeSpecName: "kube-api-access-bgcnq") pod "cba2d642-05ff-4d04-98e9-4430766adc35" (UID: "cba2d642-05ff-4d04-98e9-4430766adc35"). InnerVolumeSpecName "kube-api-access-bgcnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:28:06.089415 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.089278    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cba2d642-05ff-4d04-98e9-4430766adc35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cba2d642-05ff-4d04-98e9-4430766adc35" (UID: "cba2d642-05ff-4d04-98e9-4430766adc35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:28:06.187763 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.187733    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cba2d642-05ff-4d04-98e9-4430766adc35-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:06.187763 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.187757    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cba2d642-05ff-4d04-98e9-4430766adc35-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:06.187763 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.187769    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgcnq\" (UniqueName: \"kubernetes.io/projected/cba2d642-05ff-4d04-98e9-4430766adc35-kube-api-access-bgcnq\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:28:06.355138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.355093    2576 generic.go:356] "Generic (PLEG): container finished" podID="cba2d642-05ff-4d04-98e9-4430766adc35" containerID="1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d" exitCode=0
Sep 09 20:28:06.355138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.355137    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"
Sep 09 20:28:06.355345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.355174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" event={"ID":"cba2d642-05ff-4d04-98e9-4430766adc35","Type":"ContainerDied","Data":"1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d"}
Sep 09 20:28:06.355345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.355223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml" event={"ID":"cba2d642-05ff-4d04-98e9-4430766adc35","Type":"ContainerDied","Data":"9e2abae14b1b45e877f97ff60f80cacb5380716dd1e46673216683491174fe5f"}
Sep 09 20:28:06.355345 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.355255    2576 scope.go:117] "RemoveContainer" containerID="1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d"
Sep 09 20:28:06.356656 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.356632    2576 generic.go:356] "Generic (PLEG): container finished" podID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerID="60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4" exitCode=0
Sep 09 20:28:06.356762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.356669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" event={"ID":"3487a2ff-027c-4490-9ab2-5b6154c5e6d8","Type":"ContainerDied","Data":"60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4"}
Sep 09 20:28:06.356762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.356681    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"
Sep 09 20:28:06.356762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.356713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv" event={"ID":"3487a2ff-027c-4490-9ab2-5b6154c5e6d8","Type":"ContainerDied","Data":"86cdb22d5001264d567c3b90ff30c9756a4a4be599fae33e8a772ed0f432c7e8"}
Sep 09 20:28:06.366371 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.365219    2576 scope.go:117] "RemoveContainer" containerID="1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d"
Sep 09 20:28:06.366371 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:28:06.366217    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d\": container with ID starting with 1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d not found: ID does not exist" containerID="1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d"
Sep 09 20:28:06.366371 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.366249    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d"} err="failed to get container status \"1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d\": rpc error: code = NotFound desc = could not find container \"1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d\": container with ID starting with 1faf9ad4d49a15966242899617689d8054efeb6171d30d6444a64e6e72be946d not found: ID does not exist"
Sep 09 20:28:06.366371 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.366271    2576 scope.go:117] "RemoveContainer" containerID="60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4"
Sep 09 20:28:06.374272 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.374245    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"]
Sep 09 20:28:06.374998 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.374934    2576 scope.go:117] "RemoveContainer" containerID="60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4"
Sep 09 20:28:06.375362 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:28:06.375339    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4\": container with ID starting with 60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4 not found: ID does not exist" containerID="60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4"
Sep 09 20:28:06.375500 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.375375    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4"} err="failed to get container status \"60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4\": rpc error: code = NotFound desc = could not find container \"60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4\": container with ID starting with 60b27a6b32c2e0bff8aab3a5d950e89eeb132052f78f01a6e72f12dfa00ffde4 not found: ID does not exist"
Sep 09 20:28:06.377716 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.377696    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-router-schedz2mdv"]
Sep 09 20:28:06.388451 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.388431    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"]
Sep 09 20:28:06.392926 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:06.392905    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v1-kserve-5657b88c95-mblml"]
Sep 09 20:28:07.961695 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:07.961662    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" path="/var/lib/kubelet/pods/3487a2ff-027c-4490-9ab2-5b6154c5e6d8/volumes"
Sep 09 20:28:07.962098 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:07.962059    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cba2d642-05ff-4d04-98e9-4430766adc35" path="/var/lib/kubelet/pods/cba2d642-05ff-4d04-98e9-4430766adc35/volumes"
Sep 09 20:28:35.168828 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.168790    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"]
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169254    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cba2d642-05ff-4d04-98e9-4430766adc35" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169271    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cba2d642-05ff-4d04-98e9-4430766adc35" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169282    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1592c5a2-f5b3-479e-82b0-2cbee43d5888" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169288    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1592c5a2-f5b3-479e-82b0-2cbee43d5888" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169298    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169303    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169353    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cba2d642-05ff-4d04-98e9-4430766adc35" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169360    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1592c5a2-f5b3-479e-82b0-2cbee43d5888" containerName="main"
Sep 09 20:28:35.169473 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.169366    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3487a2ff-027c-4490-9ab2-5b6154c5e6d8" containerName="main"
Sep 09 20:28:35.172172 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.172153    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.175339 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.175316    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 09 20:28:35.176009 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.175983    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-fcrwm\""
Sep 09 20:28:35.176128 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.176047    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-jqv5b\""
Sep 09 20:28:35.176128 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.176066    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 09 20:28:35.176128 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.176111    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 09 20:28:35.195814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.195790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvjfb\" (UniqueName: \"kubernetes.io/projected/705a17cd-9aad-47a9-a0f2-3cf7557231de-kube-api-access-pvjfb\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.195907 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.195837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705a17cd-9aad-47a9-a0f2-3cf7557231de-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.201806 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.201786    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"]
Sep 09 20:28:35.297295 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.297258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705a17cd-9aad-47a9-a0f2-3cf7557231de-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.297488 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.297347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvjfb\" (UniqueName: \"kubernetes.io/projected/705a17cd-9aad-47a9-a0f2-3cf7557231de-kube-api-access-pvjfb\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.299917 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.299895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705a17cd-9aad-47a9-a0f2-3cf7557231de-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.307192 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.307167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvjfb\" (UniqueName: \"kubernetes.io/projected/705a17cd-9aad-47a9-a0f2-3cf7557231de-kube-api-access-pvjfb\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.482563 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.482467    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:35.756258 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:35.756231    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"]
Sep 09 20:28:35.758241 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:28:35.758214    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705a17cd_9aad_47a9_a0f2_3cf7557231de.slice/crio-778791ce167f60767d5c846bbdcd72ba8857865f3fc0d95e75aaece3b81e4a8d WatchSource:0}: Error finding container 778791ce167f60767d5c846bbdcd72ba8857865f3fc0d95e75aaece3b81e4a8d: Status 404 returned error can't find the container with id 778791ce167f60767d5c846bbdcd72ba8857865f3fc0d95e75aaece3b81e4a8d
Sep 09 20:28:36.500780 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:36.500739    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" event={"ID":"705a17cd-9aad-47a9-a0f2-3cf7557231de","Type":"ContainerStarted","Data":"c513c787ab61ac8cf27e7d3d5856c2451a6ce4583e504469f0c087388e889106"}
Sep 09 20:28:36.501163 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:36.500791    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" event={"ID":"705a17cd-9aad-47a9-a0f2-3cf7557231de","Type":"ContainerStarted","Data":"778791ce167f60767d5c846bbdcd72ba8857865f3fc0d95e75aaece3b81e4a8d"}
Sep 09 20:28:36.501163 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:36.500823    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:28:36.518497 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:28:36.518447    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" podStartSLOduration=1.518429019 podStartE2EDuration="1.518429019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:28:36.516573106 +0000 UTC m=+2019.034710670" watchObservedRunningTime="2026-09-09 20:28:36.518429019 +0000 UTC m=+2019.036566546"
Sep 09 20:29:07.507080 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:29:07.507047    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:29:58.067495 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:29:58.067448    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:29:58.070080 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:29:58.070054    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:30:38.150478 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:30:38.150440    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"]
Sep 09 20:30:47.508044 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:30:47.508004    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:30:53.174610 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:30:53.174550    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main" containerID="cri-o://c513c787ab61ac8cf27e7d3d5856c2451a6ce4583e504469f0c087388e889106" gracePeriod=45
Sep 09 20:30:57.507849 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:30:57.507808    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:31:07.507310 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:07.507266    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:31:07.507690 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:07.507375    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:31:17.506896 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:17.506858    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:31:23.309543 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.309499    2576 generic.go:356] "Generic (PLEG): container finished" podID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerID="c513c787ab61ac8cf27e7d3d5856c2451a6ce4583e504469f0c087388e889106" exitCode=0
Sep 09 20:31:23.309543 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.309538    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" event={"ID":"705a17cd-9aad-47a9-a0f2-3cf7557231de","Type":"ContainerDied","Data":"c513c787ab61ac8cf27e7d3d5856c2451a6ce4583e504469f0c087388e889106"}
Sep 09 20:31:23.424845 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.424819    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:31:23.569023 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.568930    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvjfb\" (UniqueName: \"kubernetes.io/projected/705a17cd-9aad-47a9-a0f2-3cf7557231de-kube-api-access-pvjfb\") pod \"705a17cd-9aad-47a9-a0f2-3cf7557231de\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") "
Sep 09 20:31:23.569023 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.568993    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705a17cd-9aad-47a9-a0f2-3cf7557231de-tls-certs\") pod \"705a17cd-9aad-47a9-a0f2-3cf7557231de\" (UID: \"705a17cd-9aad-47a9-a0f2-3cf7557231de\") "
Sep 09 20:31:23.571283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.571246    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/705a17cd-9aad-47a9-a0f2-3cf7557231de-kube-api-access-pvjfb" (OuterVolumeSpecName: "kube-api-access-pvjfb") pod "705a17cd-9aad-47a9-a0f2-3cf7557231de" (UID: "705a17cd-9aad-47a9-a0f2-3cf7557231de"). InnerVolumeSpecName "kube-api-access-pvjfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:31:23.571400 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.571364    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/705a17cd-9aad-47a9-a0f2-3cf7557231de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "705a17cd-9aad-47a9-a0f2-3cf7557231de" (UID: "705a17cd-9aad-47a9-a0f2-3cf7557231de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:31:23.669488 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.669454    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvjfb\" (UniqueName: \"kubernetes.io/projected/705a17cd-9aad-47a9-a0f2-3cf7557231de-kube-api-access-pvjfb\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:31:23.669488 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:23.669482    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/705a17cd-9aad-47a9-a0f2-3cf7557231de-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:31:24.316212 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:24.316131    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"
Sep 09 20:31:24.316663 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:24.316131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr" event={"ID":"705a17cd-9aad-47a9-a0f2-3cf7557231de","Type":"ContainerDied","Data":"778791ce167f60767d5c846bbdcd72ba8857865f3fc0d95e75aaece3b81e4a8d"}
Sep 09 20:31:24.316663 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:24.316285    2576 scope.go:117] "RemoveContainer" containerID="c513c787ab61ac8cf27e7d3d5856c2451a6ce4583e504469f0c087388e889106"
Sep 09 20:31:24.332253 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:24.332225    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"]
Sep 09 20:31:24.336007 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:24.335986    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69sthmr"]
Sep 09 20:31:25.962628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:25.962591    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" path="/var/lib/kubelet/pods/705a17cd-9aad-47a9-a0f2-3cf7557231de/volumes"
Sep 09 20:31:33.373315 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.373280    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"]
Sep 09 20:31:33.374251 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.374210    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main"
Sep 09 20:31:33.374251 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.374238    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main"
Sep 09 20:31:33.374449 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.374362    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="705a17cd-9aad-47a9-a0f2-3cf7557231de" containerName="main"
Sep 09 20:31:33.377077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.377059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.379724 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.379694    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 09 20:31:33.380485 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.380459    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-4zps6\""
Sep 09 20:31:33.380594 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.380466    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 09 20:31:33.380594 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.380575    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 09 20:31:33.380839 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.380815    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-n74bf\""
Sep 09 20:31:33.390861 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.390832    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"]
Sep 09 20:31:33.550252 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.550209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cttfz\" (UniqueName: \"kubernetes.io/projected/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-kube-api-access-cttfz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.550436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.550291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.651442 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.651353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cttfz\" (UniqueName: \"kubernetes.io/projected/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-kube-api-access-cttfz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.651442 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.651419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.654099 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.654076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.658875 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.658851    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cttfz\" (UniqueName: \"kubernetes.io/projected/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-kube-api-access-cttfz\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.688571 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.688534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:33.962430 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:31:33.962386    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232ee7e0_3cb1_42eb_af52_7dd8b4d8c9cd.slice/crio-c26e7cfafa1388f948082937821e55d85f40f9b88520bb98a0d90bb7823950cd WatchSource:0}: Error finding container c26e7cfafa1388f948082937821e55d85f40f9b88520bb98a0d90bb7823950cd: Status 404 returned error can't find the container with id c26e7cfafa1388f948082937821e55d85f40f9b88520bb98a0d90bb7823950cd
Sep 09 20:31:33.963430 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:33.963407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"]
Sep 09 20:31:34.369028 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:34.368986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" event={"ID":"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd","Type":"ContainerStarted","Data":"1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889"}
Sep 09 20:31:34.369214 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:34.369050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" event={"ID":"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd","Type":"ContainerStarted","Data":"c26e7cfafa1388f948082937821e55d85f40f9b88520bb98a0d90bb7823950cd"}
Sep 09 20:31:34.369214 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:34.369076    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:31:34.388512 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:31:34.388464    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" podStartSLOduration=1.388447363 podStartE2EDuration="1.388447363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:31:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:31:34.387162073 +0000 UTC m=+2196.905299602" watchObservedRunningTime="2026-09-09 20:31:34.388447363 +0000 UTC m=+2196.906584923"
Sep 09 20:32:05.375132 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:05.375104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:32:29.204558 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:29.204477    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"]
Sep 09 20:32:35.375886 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:35.375840    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:44.228048 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:44.228001    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main" containerID="cri-o://1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889" gracePeriod=45
Sep 09 20:32:45.375197 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:45.375156    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:55.376056 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:55.376017    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:32:55.376445 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:32:55.376129    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:33:05.375463 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:05.375422    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:33:14.474811 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.474786    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:33:14.557639 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.557548    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-tls-certs\") pod \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") "
Sep 09 20:33:14.557639 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.557594    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cttfz\" (UniqueName: \"kubernetes.io/projected/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-kube-api-access-cttfz\") pod \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\" (UID: \"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd\") "
Sep 09 20:33:14.559906 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.559878    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-kube-api-access-cttfz" (OuterVolumeSpecName: "kube-api-access-cttfz") pod "232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" (UID: "232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd"). InnerVolumeSpecName "kube-api-access-cttfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:33:14.559906 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.559896    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" (UID: "232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:33:14.658273 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.658233    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.658273 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.658270    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cttfz\" (UniqueName: \"kubernetes.io/projected/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd-kube-api-access-cttfz\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:33:14.848699 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.848662    2576 generic.go:356] "Generic (PLEG): container finished" podID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerID="1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889" exitCode=0
Sep 09 20:33:14.848869 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.848712    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"
Sep 09 20:33:14.848869 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.848750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" event={"ID":"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd","Type":"ContainerDied","Data":"1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889"}
Sep 09 20:33:14.848869 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.848805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk" event={"ID":"232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd","Type":"ContainerDied","Data":"c26e7cfafa1388f948082937821e55d85f40f9b88520bb98a0d90bb7823950cd"}
Sep 09 20:33:14.848869 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.848829    2576 scope.go:117] "RemoveContainer" containerID="1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889"
Sep 09 20:33:14.858453 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.858434    2576 scope.go:117] "RemoveContainer" containerID="1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889"
Sep 09 20:33:14.858732 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:33:14.858711    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889\": container with ID starting with 1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889 not found: ID does not exist" containerID="1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889"
Sep 09 20:33:14.858774 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.858739    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889"} err="failed to get container status \"1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889\": rpc error: code = NotFound desc = could not find container \"1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889\": container with ID starting with 1074ba4ed5059075ee8b5f41762e07e82c83032a7158e45e0e95de43758a8889 not found: ID does not exist"
Sep 09 20:33:14.870768 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.870742    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"]
Sep 09 20:33:14.874674 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:14.874652    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785j9khk"]
Sep 09 20:33:15.961723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:15.961688    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" path="/var/lib/kubelet/pods/232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd/volumes"
Sep 09 20:33:22.648274 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.648239    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"]
Sep 09 20:33:22.648742 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.648721    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main"
Sep 09 20:33:22.648811 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.648746    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main"
Sep 09 20:33:22.648855 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.648823    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="232ee7e0-3cb1-42eb-af52-7dd8b4d8c9cd" containerName="main"
Sep 09 20:33:22.653483 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.653463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.657098 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.657076    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 09 20:33:22.657280 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.657230    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 09 20:33:22.657363 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.657236    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-89pnb\""
Sep 09 20:33:22.657363 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.657322    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 09 20:33:22.657700 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.657636    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-z4j6r\""
Sep 09 20:33:22.661870 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.661847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"]
Sep 09 20:33:22.713726 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.713689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f23bc53-9e22-49d2-b725-162bec3af799-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.713896 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.713753    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxtpf\" (UniqueName: \"kubernetes.io/projected/5f23bc53-9e22-49d2-b725-162bec3af799-kube-api-access-pxtpf\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.814860 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.814823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxtpf\" (UniqueName: \"kubernetes.io/projected/5f23bc53-9e22-49d2-b725-162bec3af799-kube-api-access-pxtpf\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.815095 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.814874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f23bc53-9e22-49d2-b725-162bec3af799-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.817638 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.817606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f23bc53-9e22-49d2-b725-162bec3af799-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.822747 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.822724    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxtpf\" (UniqueName: \"kubernetes.io/projected/5f23bc53-9e22-49d2-b725-162bec3af799-kube-api-access-pxtpf\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:22.966264 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:22.966155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:23.261226 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:23.261196    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"]
Sep 09 20:33:23.263055 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:33:23.263015    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f23bc53_9e22_49d2_b725_162bec3af799.slice/crio-fa7fe12d9cb98338826b549dca8a65035699845048e79839ff060425f690dd70 WatchSource:0}: Error finding container fa7fe12d9cb98338826b549dca8a65035699845048e79839ff060425f690dd70: Status 404 returned error can't find the container with id fa7fe12d9cb98338826b549dca8a65035699845048e79839ff060425f690dd70
Sep 09 20:33:23.900270 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:23.900236    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" event={"ID":"5f23bc53-9e22-49d2-b725-162bec3af799","Type":"ContainerStarted","Data":"3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418"}
Sep 09 20:33:23.900646 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:23.900291    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" event={"ID":"5f23bc53-9e22-49d2-b725-162bec3af799","Type":"ContainerStarted","Data":"fa7fe12d9cb98338826b549dca8a65035699845048e79839ff060425f690dd70"}
Sep 09 20:33:23.900646 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:23.900321    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:33:23.919566 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:23.919508    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podStartSLOduration=1.919491047 podStartE2EDuration="1.919491047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:33:23.917247342 +0000 UTC m=+2306.435384862" watchObservedRunningTime="2026-09-09 20:33:23.919491047 +0000 UTC m=+2306.437628573"
Sep 09 20:33:54.907483 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:33:54.907450    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:34:12.864539 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:12.864503    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"]
Sep 09 20:34:14.906984 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:14.906929    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:24.906920 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:24.906882    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:27.885185 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:27.885142    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main" containerID="cri-o://3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418" gracePeriod=45
Sep 09 20:34:34.907272 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:34.907232    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:34.907654 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:34.907344    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:34:44.907045 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:44.907007    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:54.907074 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:54.907027    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:34:58.102887 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.102860    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:34:58.106834 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.106808    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:34:58.144939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.144875    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:34:58.210560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.210526    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f23bc53-9e22-49d2-b725-162bec3af799-tls-certs\") pod \"5f23bc53-9e22-49d2-b725-162bec3af799\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") "
Sep 09 20:34:58.210755 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.210591    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxtpf\" (UniqueName: \"kubernetes.io/projected/5f23bc53-9e22-49d2-b725-162bec3af799-kube-api-access-pxtpf\") pod \"5f23bc53-9e22-49d2-b725-162bec3af799\" (UID: \"5f23bc53-9e22-49d2-b725-162bec3af799\") "
Sep 09 20:34:58.213014 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.212986    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f23bc53-9e22-49d2-b725-162bec3af799-kube-api-access-pxtpf" (OuterVolumeSpecName: "kube-api-access-pxtpf") pod "5f23bc53-9e22-49d2-b725-162bec3af799" (UID: "5f23bc53-9e22-49d2-b725-162bec3af799"). InnerVolumeSpecName "kube-api-access-pxtpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:34:58.213124 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.213010    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f23bc53-9e22-49d2-b725-162bec3af799-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f23bc53-9e22-49d2-b725-162bec3af799" (UID: "5f23bc53-9e22-49d2-b725-162bec3af799"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:34:58.311432 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.311387    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxtpf\" (UniqueName: \"kubernetes.io/projected/5f23bc53-9e22-49d2-b725-162bec3af799-kube-api-access-pxtpf\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:34:58.311432 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.311425    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f23bc53-9e22-49d2-b725-162bec3af799-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:34:58.350317 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.350279    2576 generic.go:356] "Generic (PLEG): container finished" podID="5f23bc53-9e22-49d2-b725-162bec3af799" containerID="3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418" exitCode=0
Sep 09 20:34:58.350467 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.350328    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" event={"ID":"5f23bc53-9e22-49d2-b725-162bec3af799","Type":"ContainerDied","Data":"3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418"}
Sep 09 20:34:58.350467 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.350339    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"
Sep 09 20:34:58.350467 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.350412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5" event={"ID":"5f23bc53-9e22-49d2-b725-162bec3af799","Type":"ContainerDied","Data":"fa7fe12d9cb98338826b549dca8a65035699845048e79839ff060425f690dd70"}
Sep 09 20:34:58.350467 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.350442    2576 scope.go:117] "RemoveContainer" containerID="3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418"
Sep 09 20:34:58.359528 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.359513    2576 scope.go:117] "RemoveContainer" containerID="3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418"
Sep 09 20:34:58.359786 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:34:58.359768    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418\": container with ID starting with 3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418 not found: ID does not exist" containerID="3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418"
Sep 09 20:34:58.359842 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.359793    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418"} err="failed to get container status \"3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418\": rpc error: code = NotFound desc = could not find container \"3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418\": container with ID starting with 3f222e3ad05ea9a9efa391d34e81acc42ea52f1ebfb8c9bebb11f12a32f78418 not found: ID does not exist"
Sep 09 20:34:58.373487 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.373462    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"]
Sep 09 20:34:58.377458 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:58.377436    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7d6mzk5"]
Sep 09 20:34:59.962214 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:34:59.962182    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" path="/var/lib/kubelet/pods/5f23bc53-9e22-49d2-b725-162bec3af799/volumes"
Sep 09 20:35:05.634014 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.633967    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"]
Sep 09 20:35:05.634554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.634528    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main"
Sep 09 20:35:05.634695 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.634580    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main"
Sep 09 20:35:05.634762 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.634704    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f23bc53-9e22-49d2-b725-162bec3af799" containerName="main"
Sep 09 20:35:05.637484 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.637459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.641010 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.640984    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"default-dockercfg-kg8l5\""
Sep 09 20:35:05.641138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.641020    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"roll-v2-kserve-self-signed-certs\""
Sep 09 20:35:05.641138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.640988    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"kube-root-ca.crt\""
Sep 09 20:35:05.641138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.641031    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rollback-b8fdb441\"/\"openshift-service-ca.crt\""
Sep 09 20:35:05.649181 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.649158    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"]
Sep 09 20:35:05.770930 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.770892    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-home\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.770930 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.770935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqpvx\" (UniqueName: \"kubernetes.io/projected/e4986a7f-a6bb-42d0-8831-88b7320d60bc-kube-api-access-vqpvx\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.771212 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.770980    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.771212 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.771088    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-dshm\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.771212 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.771133    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.771212 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.771200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.871726 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.871688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.871726 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.871728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872025 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.871773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-home\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872025 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.871797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqpvx\" (UniqueName: \"kubernetes.io/projected/e4986a7f-a6bb-42d0-8831-88b7320d60bc-kube-api-access-vqpvx\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872025 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.871820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872025 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.871846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-dshm\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872246 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.872186    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tmp-dir\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872246 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.872223    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-model-cache\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.872246 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.872236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-home\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.874262 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.874240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-dshm\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.874510 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.874493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tls-certs\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.880879 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.880854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqpvx\" (UniqueName: \"kubernetes.io/projected/e4986a7f-a6bb-42d0-8831-88b7320d60bc-kube-api-access-vqpvx\") pod \"roll-v2-kserve-d4dbf9d67-xgw98\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") " pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:05.948192 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:05.947943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:06.227072 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:06.227046    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"]
Sep 09 20:35:06.228793 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:35:06.228765    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4986a7f_a6bb_42d0_8831_88b7320d60bc.slice/crio-84fcb707cec50530cac0d82c4ede51f128911a1a073e066521e1ad97d1387795 WatchSource:0}: Error finding container 84fcb707cec50530cac0d82c4ede51f128911a1a073e066521e1ad97d1387795: Status 404 returned error can't find the container with id 84fcb707cec50530cac0d82c4ede51f128911a1a073e066521e1ad97d1387795
Sep 09 20:35:06.393053 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:06.393010    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98" event={"ID":"e4986a7f-a6bb-42d0-8831-88b7320d60bc","Type":"ContainerStarted","Data":"086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21"}
Sep 09 20:35:06.393243 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:06.393081    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98" event={"ID":"e4986a7f-a6bb-42d0-8831-88b7320d60bc","Type":"ContainerStarted","Data":"84fcb707cec50530cac0d82c4ede51f128911a1a073e066521e1ad97d1387795"}
Sep 09 20:35:06.418900 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:06.418839    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98" podStartSLOduration=1.418815798 podStartE2EDuration="1.418815798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:06.415892703 +0000 UTC m=+2408.934030231" watchObservedRunningTime="2026-09-09 20:35:06.418815798 +0000 UTC m=+2408.936953329"
Sep 09 20:35:06.948590 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:06.948553    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:15.949012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:15.948976    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:15.963413 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:15.963388    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:16.454033 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:16.454004    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:35:31.996413 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:31.996337    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"]
Sep 09 20:35:32.000095 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.000071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.003476 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.003449    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 09 20:35:32.004156 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.004134    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 09 20:35:32.004472 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.004450    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-c4v65\""
Sep 09 20:35:32.004594 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.004494    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 09 20:35:32.004594 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.004501    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-n5lvh\""
Sep 09 20:35:32.011660 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.011640    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"]
Sep 09 20:35:32.088307 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.088275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcls\" (UniqueName: \"kubernetes.io/projected/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-kube-api-access-5rcls\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.088499 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.088338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.188898 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.188847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcls\" (UniqueName: \"kubernetes.io/projected/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-kube-api-access-5rcls\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.189097 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.188920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.191657 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.191631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.199847 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.199822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcls\" (UniqueName: \"kubernetes.io/projected/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-kube-api-access-5rcls\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.311694 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.311607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:32.591780 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:32.591734    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"]
Sep 09 20:35:32.594053 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:35:32.594027    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bebed9e_f63b_47eb_80b9_6b750cfa2f68.slice/crio-f217d6cccc152370dd27dbe20ac5749a7176f90e7feae0e5d9bddeb4adc36bf7 WatchSource:0}: Error finding container f217d6cccc152370dd27dbe20ac5749a7176f90e7feae0e5d9bddeb4adc36bf7: Status 404 returned error can't find the container with id f217d6cccc152370dd27dbe20ac5749a7176f90e7feae0e5d9bddeb4adc36bf7
Sep 09 20:35:33.525904 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:33.525868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" event={"ID":"3bebed9e-f63b-47eb-80b9-6b750cfa2f68","Type":"ContainerStarted","Data":"640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89"}
Sep 09 20:35:33.526283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:33.525926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" event={"ID":"3bebed9e-f63b-47eb-80b9-6b750cfa2f68","Type":"ContainerStarted","Data":"f217d6cccc152370dd27dbe20ac5749a7176f90e7feae0e5d9bddeb4adc36bf7"}
Sep 09 20:35:33.526283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:33.525974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:35:33.546823 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:35:33.546773    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" podStartSLOduration=2.546758609 podStartE2EDuration="2.546758609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:35:33.544682197 +0000 UTC m=+2436.062819724" watchObservedRunningTime="2026-09-09 20:35:33.546758609 +0000 UTC m=+2436.064896135"
Sep 09 20:36:04.533702 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:04.533543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:36:09.151470 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:09.151436    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"]
Sep 09 20:36:14.533131 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:14.533096    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:36:24.175289 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:24.175248    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main" containerID="cri-o://640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89" gracePeriod=45
Sep 09 20:36:24.532550 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:24.532457    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:36:34.532226 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:34.532190    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:36:34.532674 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:34.532309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:36:44.533143 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:44.533102    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:36:46.117619 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:46.117576    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"]
Sep 09 20:36:54.421009 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.420984    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:36:54.439468 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.439395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5rcls\" (UniqueName: \"kubernetes.io/projected/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-kube-api-access-5rcls\") pod \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") "
Sep 09 20:36:54.439607 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.439467    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-tls-certs\") pod \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\" (UID: \"3bebed9e-f63b-47eb-80b9-6b750cfa2f68\") "
Sep 09 20:36:54.441808 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.441776    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-kube-api-access-5rcls" (OuterVolumeSpecName: "kube-api-access-5rcls") pod "3bebed9e-f63b-47eb-80b9-6b750cfa2f68" (UID: "3bebed9e-f63b-47eb-80b9-6b750cfa2f68"). InnerVolumeSpecName "kube-api-access-5rcls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:36:54.441944 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.441877    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3bebed9e-f63b-47eb-80b9-6b750cfa2f68" (UID: "3bebed9e-f63b-47eb-80b9-6b750cfa2f68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:36:54.540656 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.540618    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:36:54.540656 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.540649    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rcls\" (UniqueName: \"kubernetes.io/projected/3bebed9e-f63b-47eb-80b9-6b750cfa2f68-kube-api-access-5rcls\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:36:54.924775 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.924740    2576 generic.go:356] "Generic (PLEG): container finished" podID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerID="640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89" exitCode=0
Sep 09 20:36:54.924949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.924783    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"
Sep 09 20:36:54.924949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.924800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" event={"ID":"3bebed9e-f63b-47eb-80b9-6b750cfa2f68","Type":"ContainerDied","Data":"640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89"}
Sep 09 20:36:54.924949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.924850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz" event={"ID":"3bebed9e-f63b-47eb-80b9-6b750cfa2f68","Type":"ContainerDied","Data":"f217d6cccc152370dd27dbe20ac5749a7176f90e7feae0e5d9bddeb4adc36bf7"}
Sep 09 20:36:54.924949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.924875    2576 scope.go:117] "RemoveContainer" containerID="640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89"
Sep 09 20:36:54.936420 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.936394    2576 scope.go:117] "RemoveContainer" containerID="640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89"
Sep 09 20:36:54.936880 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:36:54.936853    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89\": container with ID starting with 640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89 not found: ID does not exist" containerID="640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89"
Sep 09 20:36:54.936987 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.936891    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89"} err="failed to get container status \"640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89\": rpc error: code = NotFound desc = could not find container \"640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89\": container with ID starting with 640be7d1fbb5363963814a9b0726f7f4b49d04dedc945d91666525d3fc152a89 not found: ID does not exist"
Sep 09 20:36:54.948541 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.948513    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"]
Sep 09 20:36:54.953834 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:54.953808    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schelmjsz"]
Sep 09 20:36:55.962500 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:36:55.962468    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" path="/var/lib/kubelet/pods/3bebed9e-f63b-47eb-80b9-6b750cfa2f68/volumes"
Sep 09 20:37:00.325177 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.325141    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"]
Sep 09 20:37:00.325628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.325493    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main"
Sep 09 20:37:00.325628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.325514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main"
Sep 09 20:37:00.325628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.325566    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bebed9e-f63b-47eb-80b9-6b750cfa2f68" containerName="main"
Sep 09 20:37:00.328910 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.328844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.331550 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.331523    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"kube-root-ca.crt\""
Sep 09 20:37:00.331700 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.331571    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"openshift-service-ca.crt\""
Sep 09 20:37:00.332486 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.332469    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"default-dockercfg-d9xsv\""
Sep 09 20:37:00.332566 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.332551    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-route-owner-5f630121\"/\"owner-v1-kserve-self-signed-certs\""
Sep 09 20:37:00.342701 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.342668    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"]
Sep 09 20:37:00.390674 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.390636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.390674 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.390683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-model-cache\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.390932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.390714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-home\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.390932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.390768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-dshm\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.390932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.390783    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.390932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.390881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htlb\" (UniqueName: \"kubernetes.io/projected/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-kube-api-access-2htlb\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.491946 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.491905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htlb\" (UniqueName: \"kubernetes.io/projected/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-kube-api-access-2htlb\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.491946 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.491947    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-model-cache\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-home\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492235 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-dshm\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492377 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492298    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492459 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tmp-dir\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492520 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-model-cache\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.492520 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.492504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-home\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.494713 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.494687    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-dshm\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.495050 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.495028    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tls-certs\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.500870 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.500833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htlb\" (UniqueName: \"kubernetes.io/projected/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-kube-api-access-2htlb\") pod \"owner-v1-kserve-75579ff5bc-qxg6l\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") " pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.639921 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.639879    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:00.972662 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:00.972635    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"]
Sep 09 20:37:00.975317 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:37:00.975279    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf97aea7_2b9d_4af4_b2b0_aaf2a677fea0.slice/crio-c522945cac896ecf259c35374e1bc1806415d84932ffe1f9dd1ba706667ff582 WatchSource:0}: Error finding container c522945cac896ecf259c35374e1bc1806415d84932ffe1f9dd1ba706667ff582: Status 404 returned error can't find the container with id c522945cac896ecf259c35374e1bc1806415d84932ffe1f9dd1ba706667ff582
Sep 09 20:37:01.138031 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.137986    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98" podUID="e4986a7f-a6bb-42d0-8831-88b7320d60bc" containerName="main" containerID="cri-o://086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21" gracePeriod=45
Sep 09 20:37:01.384810 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.384772    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:37:01.500105 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500073    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqpvx\" (UniqueName: \"kubernetes.io/projected/e4986a7f-a6bb-42d0-8831-88b7320d60bc-kube-api-access-vqpvx\") pod \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") "
Sep 09 20:37:01.500282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500177    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-dshm\") pod \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") "
Sep 09 20:37:01.500282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500201    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tmp-dir\") pod \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") "
Sep 09 20:37:01.500282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500243    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tls-certs\") pod \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") "
Sep 09 20:37:01.500282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500280    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-home\") pod \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") "
Sep 09 20:37:01.500518 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500303    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-model-cache\") pod \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\" (UID: \"e4986a7f-a6bb-42d0-8831-88b7320d60bc\") "
Sep 09 20:37:01.500935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500559    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4986a7f-a6bb-42d0-8831-88b7320d60bc" (UID: "e4986a7f-a6bb-42d0-8831-88b7320d60bc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:01.500935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500681    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-home" (OuterVolumeSpecName: "home") pod "e4986a7f-a6bb-42d0-8831-88b7320d60bc" (UID: "e4986a7f-a6bb-42d0-8831-88b7320d60bc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:01.500935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.500752    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4986a7f-a6bb-42d0-8831-88b7320d60bc" (UID: "e4986a7f-a6bb-42d0-8831-88b7320d60bc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:01.503100 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.503039    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-dshm" (OuterVolumeSpecName: "dshm") pod "e4986a7f-a6bb-42d0-8831-88b7320d60bc" (UID: "e4986a7f-a6bb-42d0-8831-88b7320d60bc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:37:01.503100 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.503049    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4986a7f-a6bb-42d0-8831-88b7320d60bc-kube-api-access-vqpvx" (OuterVolumeSpecName: "kube-api-access-vqpvx") pod "e4986a7f-a6bb-42d0-8831-88b7320d60bc" (UID: "e4986a7f-a6bb-42d0-8831-88b7320d60bc"). InnerVolumeSpecName "kube-api-access-vqpvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:37:01.503272 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.503170    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4986a7f-a6bb-42d0-8831-88b7320d60bc" (UID: "e4986a7f-a6bb-42d0-8831-88b7320d60bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:37:01.601827 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.601779    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.601827 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.601815    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.601827 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.601823    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.601827 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.601831    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqpvx\" (UniqueName: \"kubernetes.io/projected/e4986a7f-a6bb-42d0-8831-88b7320d60bc-kube-api-access-vqpvx\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.601827 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.601840    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.602174 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.601848    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4986a7f-a6bb-42d0-8831-88b7320d60bc-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:37:01.961325 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.961289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l" event={"ID":"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0","Type":"ContainerStarted","Data":"457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3"}
Sep 09 20:37:01.961516 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.961340    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l" event={"ID":"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0","Type":"ContainerStarted","Data":"c522945cac896ecf259c35374e1bc1806415d84932ffe1f9dd1ba706667ff582"}
Sep 09 20:37:01.962063 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.962035    2576 generic.go:356] "Generic (PLEG): container finished" podID="e4986a7f-a6bb-42d0-8831-88b7320d60bc" containerID="086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21" exitCode=0
Sep 09 20:37:01.962063 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.962063    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"
Sep 09 20:37:01.962188 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.962097    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98" event={"ID":"e4986a7f-a6bb-42d0-8831-88b7320d60bc","Type":"ContainerDied","Data":"086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21"}
Sep 09 20:37:01.962188 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.962143    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98" event={"ID":"e4986a7f-a6bb-42d0-8831-88b7320d60bc","Type":"ContainerDied","Data":"84fcb707cec50530cac0d82c4ede51f128911a1a073e066521e1ad97d1387795"}
Sep 09 20:37:01.962188 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.962174    2576 scope.go:117] "RemoveContainer" containerID="086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21"
Sep 09 20:37:01.976219 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.976188    2576 scope.go:117] "RemoveContainer" containerID="086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21"
Sep 09 20:37:01.977565 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:37:01.976639    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21\": container with ID starting with 086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21 not found: ID does not exist" containerID="086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21"
Sep 09 20:37:01.977565 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.976678    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21"} err="failed to get container status \"086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21\": rpc error: code = NotFound desc = could not find container \"086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21\": container with ID starting with 086b8ef0538d260c831a181e6e1ed0d8f8d15c84c4631e788727840979f81a21 not found: ID does not exist"
Sep 09 20:37:01.980513 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.980466    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l" podStartSLOduration=1.9804516749999999 podStartE2EDuration="1.980451675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:01.979359954 +0000 UTC m=+2524.497497482" watchObservedRunningTime="2026-09-09 20:37:01.980451675 +0000 UTC m=+2524.498589287"
Sep 09 20:37:01.994670 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.994643    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"]
Sep 09 20:37:01.998267 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:01.998234    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rollback-b8fdb441/roll-v2-kserve-d4dbf9d67-xgw98"]
Sep 09 20:37:02.640790 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:02.640744    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:03.962619 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:03.962584    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4986a7f-a6bb-42d0-8831-88b7320d60bc" path="/var/lib/kubelet/pods/e4986a7f-a6bb-42d0-8831-88b7320d60bc/volumes"
Sep 09 20:37:10.640559 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:10.640521    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:10.653257 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:10.653228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:11.017616 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:11.017526    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:37:24.179727 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.179695    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"]
Sep 09 20:37:24.180238 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.180032    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4986a7f-a6bb-42d0-8831-88b7320d60bc" containerName="main"
Sep 09 20:37:24.180238 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.180045    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4986a7f-a6bb-42d0-8831-88b7320d60bc" containerName="main"
Sep 09 20:37:24.180238 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.180105    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4986a7f-a6bb-42d0-8831-88b7320d60bc" containerName="main"
Sep 09 20:37:24.183237 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.183217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.186208 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.186190    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-router-managed-test-llm-5b1e8f15-epp-sa-dockercfg-znxzx\""
Sep 09 20:37:24.186331 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.186248    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 09 20:37:24.186900 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.186865    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-l6wdn\""
Sep 09 20:37:24.187029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.186905    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 09 20:37:24.187241 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.187216    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 09 20:37:24.192645 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.192623    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"]
Sep 09 20:37:24.291147 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.291113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xlmg\" (UniqueName: \"kubernetes.io/projected/f8cda825-8d71-453b-9fe0-e90073b66501-kube-api-access-7xlmg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.291325 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.291171    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cda825-8d71-453b-9fe0-e90073b66501-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.391574 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.391536    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xlmg\" (UniqueName: \"kubernetes.io/projected/f8cda825-8d71-453b-9fe0-e90073b66501-kube-api-access-7xlmg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.391748 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.391611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cda825-8d71-453b-9fe0-e90073b66501-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.394493 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.394468    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cda825-8d71-453b-9fe0-e90073b66501-tls-certs\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.402241 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.402207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xlmg\" (UniqueName: \"kubernetes.io/projected/f8cda825-8d71-453b-9fe0-e90073b66501-kube-api-access-7xlmg\") pod \"llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") " pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.495182 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.495095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:24.776136 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:24.776107    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"]
Sep 09 20:37:24.778615 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:37:24.778587    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8cda825_8d71_453b_9fe0_e90073b66501.slice/crio-b74c5ee25929ede688a56e8eb8e212c8639265b33f05961776d4aa8bb5b21198 WatchSource:0}: Error finding container b74c5ee25929ede688a56e8eb8e212c8639265b33f05961776d4aa8bb5b21198: Status 404 returned error can't find the container with id b74c5ee25929ede688a56e8eb8e212c8639265b33f05961776d4aa8bb5b21198
Sep 09 20:37:25.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:25.076510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" event={"ID":"f8cda825-8d71-453b-9fe0-e90073b66501","Type":"ContainerStarted","Data":"2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8"}
Sep 09 20:37:25.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:25.076577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" event={"ID":"f8cda825-8d71-453b-9fe0-e90073b66501","Type":"ContainerStarted","Data":"b74c5ee25929ede688a56e8eb8e212c8639265b33f05961776d4aa8bb5b21198"}
Sep 09 20:37:25.076620 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:25.076608    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:25.094033 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:25.093939    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podStartSLOduration=1.093924652 podStartE2EDuration="1.093924652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:37:25.093284769 +0000 UTC m=+2547.611422296" watchObservedRunningTime="2026-09-09 20:37:25.093924652 +0000 UTC m=+2547.612062179"
Sep 09 20:37:56.083052 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:56.083022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:37:59.225003 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:37:59.224943    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"]
Sep 09 20:38:14.250545 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.250481    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l" podUID="df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" containerName="main" containerID="cri-o://457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3" gracePeriod=45
Sep 09 20:38:14.279478 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:38:14.279448    2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf97aea7_2b9d_4af4_b2b0_aaf2a677fea0.slice/crio-457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3.scope\": RecentStats: unable to find data in memory cache]"
Sep 09 20:38:14.498704 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.498679    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:38:14.588753 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.588671    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2htlb\" (UniqueName: \"kubernetes.io/projected/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-kube-api-access-2htlb\") pod \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") "
Sep 09 20:38:14.588753 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.588732    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tmp-dir\") pod \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") "
Sep 09 20:38:14.588753 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.588752    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-dshm\") pod \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") "
Sep 09 20:38:14.589067 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.588780    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-home\") pod \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") "
Sep 09 20:38:14.589067 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.588803    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-model-cache\") pod \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") "
Sep 09 20:38:14.589067 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.588826    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tls-certs\") pod \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\" (UID: \"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0\") "
Sep 09 20:38:14.589227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.589180    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-model-cache" (OuterVolumeSpecName: "model-cache") pod "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" (UID: "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:14.589227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.589208    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-home" (OuterVolumeSpecName: "home") pod "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" (UID: "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:14.589323 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.589229    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" (UID: "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:14.591257 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.591228    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-dshm" (OuterVolumeSpecName: "dshm") pod "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" (UID: "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:38:14.591376 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.591261    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" (UID: "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:38:14.591376 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.591289    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-kube-api-access-2htlb" (OuterVolumeSpecName: "kube-api-access-2htlb") pod "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" (UID: "df97aea7-2b9d-4af4-b2b0-aaf2a677fea0"). InnerVolumeSpecName "kube-api-access-2htlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:38:14.689378 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.689341    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htlb\" (UniqueName: \"kubernetes.io/projected/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-kube-api-access-2htlb\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:38:14.689560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.689396    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:38:14.689560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.689408    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:38:14.689560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.689415    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:38:14.689560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.689423    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:38:14.689560 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:14.689430    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:38:15.323352 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.323316    2576 generic.go:356] "Generic (PLEG): container finished" podID="df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" containerID="457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3" exitCode=0
Sep 09 20:38:15.323814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.323364    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"
Sep 09 20:38:15.323814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.323373    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l" event={"ID":"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0","Type":"ContainerDied","Data":"457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3"}
Sep 09 20:38:15.323814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.323414    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l" event={"ID":"df97aea7-2b9d-4af4-b2b0-aaf2a677fea0","Type":"ContainerDied","Data":"c522945cac896ecf259c35374e1bc1806415d84932ffe1f9dd1ba706667ff582"}
Sep 09 20:38:15.323814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.323435    2576 scope.go:117] "RemoveContainer" containerID="457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3"
Sep 09 20:38:15.333105 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.333084    2576 scope.go:117] "RemoveContainer" containerID="457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3"
Sep 09 20:38:15.333372 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:38:15.333354    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3\": container with ID starting with 457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3 not found: ID does not exist" containerID="457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3"
Sep 09 20:38:15.333416 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.333381    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3"} err="failed to get container status \"457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3\": rpc error: code = NotFound desc = could not find container \"457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3\": container with ID starting with 457c8fbc80c8bb64768c2c3fb9c44aa41f9bfd35cd94c203efca075f55c5b9e3 not found: ID does not exist"
Sep 09 20:38:15.347329 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.347304    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"]
Sep 09 20:38:15.352842 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.352820    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-route-owner-5f630121/owner-v1-kserve-75579ff5bc-qxg6l"]
Sep 09 20:38:15.961820 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:15.961789    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" path="/var/lib/kubelet/pods/df97aea7-2b9d-4af4-b2b0-aaf2a677fea0/volumes"
Sep 09 20:38:40.116805 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.116771    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"]
Sep 09 20:38:40.117293 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.117128    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" containerName="main"
Sep 09 20:38:40.117293 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.117142    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" containerName="main"
Sep 09 20:38:40.117293 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.117193    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="df97aea7-2b9d-4af4-b2b0-aaf2a677fea0" containerName="main"
Sep 09 20:38:40.119939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.119918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.123645 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.123610    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 09 20:38:40.123645 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.123610    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-ls5pt\""
Sep 09 20:38:40.130357 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.130331    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"]
Sep 09 20:38:40.187861 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.187826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/101f1cab-35aa-4a5f-b831-bd25c777e9cd-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.188039 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.187890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcr6\" (UniqueName: \"kubernetes.io/projected/101f1cab-35aa-4a5f-b831-bd25c777e9cd-kube-api-access-5fcr6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.288883 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.288837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcr6\" (UniqueName: \"kubernetes.io/projected/101f1cab-35aa-4a5f-b831-bd25c777e9cd-kube-api-access-5fcr6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.289069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.288900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/101f1cab-35aa-4a5f-b831-bd25c777e9cd-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.291725 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.291693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/101f1cab-35aa-4a5f-b831-bd25c777e9cd-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.298219 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.298192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcr6\" (UniqueName: \"kubernetes.io/projected/101f1cab-35aa-4a5f-b831-bd25c777e9cd-kube-api-access-5fcr6\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.433108 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.433021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:40.706440 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:40.706415    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"]
Sep 09 20:38:40.708702 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:38:40.708671    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101f1cab_35aa_4a5f_b831_bd25c777e9cd.slice/crio-c89da95e5c73bcfc4b97e946216cab5f57f063fcbd44fb3b2d7f0dd4a644bb5f WatchSource:0}: Error finding container c89da95e5c73bcfc4b97e946216cab5f57f063fcbd44fb3b2d7f0dd4a644bb5f: Status 404 returned error can't find the container with id c89da95e5c73bcfc4b97e946216cab5f57f063fcbd44fb3b2d7f0dd4a644bb5f
Sep 09 20:38:41.455608 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:41.455570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" event={"ID":"101f1cab-35aa-4a5f-b831-bd25c777e9cd","Type":"ContainerStarted","Data":"2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2"}
Sep 09 20:38:41.456090 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:41.455630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" event={"ID":"101f1cab-35aa-4a5f-b831-bd25c777e9cd","Type":"ContainerStarted","Data":"c89da95e5c73bcfc4b97e946216cab5f57f063fcbd44fb3b2d7f0dd4a644bb5f"}
Sep 09 20:38:41.456090 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:41.455780    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:38:41.474016 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:41.473940    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" podStartSLOduration=1.473924537 podStartE2EDuration="1.473924537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:41.472217749 +0000 UTC m=+2623.990355279" watchObservedRunningTime="2026-09-09 20:38:41.473924537 +0000 UTC m=+2623.992062063"
Sep 09 20:38:51.720424 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.720380    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"]
Sep 09 20:38:51.723842 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.723808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.727262 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.727236    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"kube-root-ca.crt\""
Sep 09 20:38:51.727262 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.727252    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"openshift-service-ca.crt\""
Sep 09 20:38:51.727515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.727236    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"default-dockercfg-2x75f\""
Sep 09 20:38:51.727515 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.727239    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-6f0aa991\"/\"llmisv122f03714c5bdf915a2917fdf1262b98-kserve-self-signed-certs\""
Sep 09 20:38:51.734411 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.734388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"]
Sep 09 20:38:51.789851 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.789811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.790077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.789929    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.790077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.790000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.790077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.790027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.790077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.790062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.790283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.790127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcnp\" (UniqueName: \"kubernetes.io/projected/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-kube-api-access-ljcnp\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.890637 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.890586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcnp\" (UniqueName: \"kubernetes.io/projected/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-kube-api-access-ljcnp\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.890820 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.890643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.890820 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.890697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.890820 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.890730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.890820 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.890765    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.891076 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.890819    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.891153 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.891129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.891220 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.891175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-home\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.891277 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.891246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.893282 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.893262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-dshm\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.893538 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.893520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:51.898070 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:51.898048    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcnp\" (UniqueName: \"kubernetes.io/projected/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-kube-api-access-ljcnp\") pod \"llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") " pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:52.036745 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:52.036646    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:38:52.313831 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:52.313798    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"]
Sep 09 20:38:52.315779 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:38:52.315734    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9351fc59_169a_43d9_99cd_fb6fc7eebbd3.slice/crio-16f621622a898cfc491ffda45789804737980d893c2716846ae83621076f448a WatchSource:0}: Error finding container 16f621622a898cfc491ffda45789804737980d893c2716846ae83621076f448a: Status 404 returned error can't find the container with id 16f621622a898cfc491ffda45789804737980d893c2716846ae83621076f448a
Sep 09 20:38:52.317852 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:52.317836    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:38:52.512849 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:52.512817    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" event={"ID":"9351fc59-169a-43d9-99cd-fb6fc7eebbd3","Type":"ContainerStarted","Data":"16f621622a898cfc491ffda45789804737980d893c2716846ae83621076f448a"}
Sep 09 20:38:54.524686 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:54.524648    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" event={"ID":"9351fc59-169a-43d9-99cd-fb6fc7eebbd3","Type":"ContainerStarted","Data":"c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"}
Sep 09 20:38:54.542547 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:54.542501    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" podStartSLOduration=1.9270478020000001 podStartE2EDuration="3.542485405s" totalImagesPullingTime="1.615437603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:38:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:38:54.541031183 +0000 UTC m=+2637.059168701" watchObservedRunningTime="2026-09-09 20:38:54.542485405 +0000 UTC m=+2637.060622959"
Sep 09 20:38:55.037654 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:38:55.037617    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:39:02.037814 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:02.037771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:39:02.050636 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:02.050606    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:39:02.581227 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:02.581199    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:39:12.462043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:12.462011    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:39:14.185324 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:14.185292    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"]
Sep 09 20:39:22.462358 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:22.462314    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:29.205456 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:29.205409    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main" containerID="cri-o://2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2" gracePeriod=45
Sep 09 20:39:32.461498 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:32.461459    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:42.462397 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:42.462357    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:42.462801 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:42.462458    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:39:52.461995 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:52.461928    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:39:58.139397 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:58.139367    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:39:58.144314 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:58.144293    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:39:59.453833 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.453810    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:39:59.560754 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.560653    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5fcr6\" (UniqueName: \"kubernetes.io/projected/101f1cab-35aa-4a5f-b831-bd25c777e9cd-kube-api-access-5fcr6\") pod \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") "
Sep 09 20:39:59.560754 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.560714    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/101f1cab-35aa-4a5f-b831-bd25c777e9cd-tls-certs\") pod \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\" (UID: \"101f1cab-35aa-4a5f-b831-bd25c777e9cd\") "
Sep 09 20:39:59.563144 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.563114    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101f1cab-35aa-4a5f-b831-bd25c777e9cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "101f1cab-35aa-4a5f-b831-bd25c777e9cd" (UID: "101f1cab-35aa-4a5f-b831-bd25c777e9cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:39:59.563144 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.563126    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101f1cab-35aa-4a5f-b831-bd25c777e9cd-kube-api-access-5fcr6" (OuterVolumeSpecName: "kube-api-access-5fcr6") pod "101f1cab-35aa-4a5f-b831-bd25c777e9cd" (UID: "101f1cab-35aa-4a5f-b831-bd25c777e9cd"). InnerVolumeSpecName "kube-api-access-5fcr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:39:59.662158 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.662121    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fcr6\" (UniqueName: \"kubernetes.io/projected/101f1cab-35aa-4a5f-b831-bd25c777e9cd-kube-api-access-5fcr6\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:39:59.662340 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.662172    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/101f1cab-35aa-4a5f-b831-bd25c777e9cd-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:39:59.850600 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.850571    2576 generic.go:356] "Generic (PLEG): container finished" podID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerID="2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2" exitCode=0
Sep 09 20:39:59.850771 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.850614    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"
Sep 09 20:39:59.850771 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.850657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" event={"ID":"101f1cab-35aa-4a5f-b831-bd25c777e9cd","Type":"ContainerDied","Data":"2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2"}
Sep 09 20:39:59.850771 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.850705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh" event={"ID":"101f1cab-35aa-4a5f-b831-bd25c777e9cd","Type":"ContainerDied","Data":"c89da95e5c73bcfc4b97e946216cab5f57f063fcbd44fb3b2d7f0dd4a644bb5f"}
Sep 09 20:39:59.850771 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.850727    2576 scope.go:117] "RemoveContainer" containerID="2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2"
Sep 09 20:39:59.859915 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.859899    2576 scope.go:117] "RemoveContainer" containerID="2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2"
Sep 09 20:39:59.860143 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:39:59.860126    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2\": container with ID starting with 2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2 not found: ID does not exist" containerID="2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2"
Sep 09 20:39:59.860198 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.860151    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2"} err="failed to get container status \"2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2\": rpc error: code = NotFound desc = could not find container \"2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2\": container with ID starting with 2c807a665b83ae169fc1536579d7c0347703b5d5088e33ae3cc18b39ca8641d2 not found: ID does not exist"
Sep 09 20:39:59.871374 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.871347    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"]
Sep 09 20:39:59.874301 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.874281    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-schemsnxh"]
Sep 09 20:39:59.961685 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:39:59.961654    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" path="/var/lib/kubelet/pods/101f1cab-35aa-4a5f-b831-bd25c777e9cd/volumes"
Sep 09 20:40:04.330127 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:04.330091    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"]
Sep 09 20:40:06.083274 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:06.083239    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:40:08.142923 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.142890    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"]
Sep 09 20:40:08.152052 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:40:08.152017    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:40:08.152052 ip-10-0-135-242 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:08.152052 ip-10-0-135-242 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:08.152052 ip-10-0-135-242 kubenswrapper[2576]:  > containerID="c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f" cmd=["/bin/sleep","15"]
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.152075    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" message="" err=<
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]:  >
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:40:08.152094    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" podUID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" containerName="main" containerID="cri-o://c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"
Sep 09 20:40:08.152349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.152135    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" podUID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" containerName="main" containerID="cri-o://c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f" gracePeriod=60
Sep 09 20:40:08.513005 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.512976    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:40:08.634639 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-home\") pod \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") "
Sep 09 20:40:08.634639 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634648    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-dshm\") pod \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") "
Sep 09 20:40:08.634882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634669    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tls-certs\") pod \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") "
Sep 09 20:40:08.634882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634694    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tmp-dir\") pod \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") "
Sep 09 20:40:08.634882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634711    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-model-cache\") pod \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") "
Sep 09 20:40:08.634882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634743    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljcnp\" (UniqueName: \"kubernetes.io/projected/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-kube-api-access-ljcnp\") pod \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\" (UID: \"9351fc59-169a-43d9-99cd-fb6fc7eebbd3\") "
Sep 09 20:40:08.635139 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.634949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-home" (OuterVolumeSpecName: "home") pod "9351fc59-169a-43d9-99cd-fb6fc7eebbd3" (UID: "9351fc59-169a-43d9-99cd-fb6fc7eebbd3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:08.635139 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.635001    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9351fc59-169a-43d9-99cd-fb6fc7eebbd3" (UID: "9351fc59-169a-43d9-99cd-fb6fc7eebbd3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:08.635139 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.635069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-model-cache" (OuterVolumeSpecName: "model-cache") pod "9351fc59-169a-43d9-99cd-fb6fc7eebbd3" (UID: "9351fc59-169a-43d9-99cd-fb6fc7eebbd3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:08.637147 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.637116    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-dshm" (OuterVolumeSpecName: "dshm") pod "9351fc59-169a-43d9-99cd-fb6fc7eebbd3" (UID: "9351fc59-169a-43d9-99cd-fb6fc7eebbd3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:40:08.637147 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.637121    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-kube-api-access-ljcnp" (OuterVolumeSpecName: "kube-api-access-ljcnp") pod "9351fc59-169a-43d9-99cd-fb6fc7eebbd3" (UID: "9351fc59-169a-43d9-99cd-fb6fc7eebbd3"). InnerVolumeSpecName "kube-api-access-ljcnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:08.637373 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.637349    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9351fc59-169a-43d9-99cd-fb6fc7eebbd3" (UID: "9351fc59-169a-43d9-99cd-fb6fc7eebbd3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:08.736012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.735902    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:08.736012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.735931    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:08.736012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.735942    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:08.736012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.735964    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:08.736012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.735973    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:08.736012 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.735980    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljcnp\" (UniqueName: \"kubernetes.io/projected/9351fc59-169a-43d9-99cd-fb6fc7eebbd3-kube-api-access-ljcnp\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:08.896006 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.895942    2576 generic.go:356] "Generic (PLEG): container finished" podID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" containerID="c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f" exitCode=0
Sep 09 20:40:08.896006 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.896001    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"
Sep 09 20:40:08.896219 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.896058    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" event={"ID":"9351fc59-169a-43d9-99cd-fb6fc7eebbd3","Type":"ContainerDied","Data":"c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"}
Sep 09 20:40:08.896219 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.896131    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k" event={"ID":"9351fc59-169a-43d9-99cd-fb6fc7eebbd3","Type":"ContainerDied","Data":"16f621622a898cfc491ffda45789804737980d893c2716846ae83621076f448a"}
Sep 09 20:40:08.896219 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.896153    2576 scope.go:117] "RemoveContainer" containerID="c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"
Sep 09 20:40:08.905292 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.905273    2576 scope.go:117] "RemoveContainer" containerID="c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"
Sep 09 20:40:08.905576 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:40:08.905554    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f\": container with ID starting with c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f not found: ID does not exist" containerID="c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"
Sep 09 20:40:08.905628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.905588    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f"} err="failed to get container status \"c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f\": rpc error: code = NotFound desc = could not find container \"c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f\": container with ID starting with c9fc54b28cf4098a3f7414d8ac33b7a846ee8809e50138239f8cf3520573ff3f not found: ID does not exist"
Sep 09 20:40:08.917933 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.917904    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"]
Sep 09 20:40:08.920065 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:08.920044    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-6f0aa991/llmisvc-model-fb-opt-125m-route-e95b1dc1-kserve-5d6ff58997bd79k"]
Sep 09 20:40:09.961390 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:09.961348    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" path="/var/lib/kubelet/pods/9351fc59-169a-43d9-99cd-fb6fc7eebbd3/volumes"
Sep 09 20:40:16.082529 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:16.082487    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:40:19.351698 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:19.351638    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main" containerID="cri-o://2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8" gracePeriod=45
Sep 09 20:40:26.083159 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:26.083121    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:40:26.083670 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:26.083251    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:40:36.082874 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:36.082834    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:40:46.083230 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:46.083186    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:40:49.605185 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.605153    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:40:49.644184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.644110    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cda825-8d71-453b-9fe0-e90073b66501-tls-certs\") pod \"f8cda825-8d71-453b-9fe0-e90073b66501\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") "
Sep 09 20:40:49.644320 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.644194    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xlmg\" (UniqueName: \"kubernetes.io/projected/f8cda825-8d71-453b-9fe0-e90073b66501-kube-api-access-7xlmg\") pod \"f8cda825-8d71-453b-9fe0-e90073b66501\" (UID: \"f8cda825-8d71-453b-9fe0-e90073b66501\") "
Sep 09 20:40:49.646203 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.646168    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8cda825-8d71-453b-9fe0-e90073b66501-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f8cda825-8d71-453b-9fe0-e90073b66501" (UID: "f8cda825-8d71-453b-9fe0-e90073b66501"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:40:49.646324 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.646246    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8cda825-8d71-453b-9fe0-e90073b66501-kube-api-access-7xlmg" (OuterVolumeSpecName: "kube-api-access-7xlmg") pod "f8cda825-8d71-453b-9fe0-e90073b66501" (UID: "f8cda825-8d71-453b-9fe0-e90073b66501"). InnerVolumeSpecName "kube-api-access-7xlmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:40:49.745460 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.745405    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xlmg\" (UniqueName: \"kubernetes.io/projected/f8cda825-8d71-453b-9fe0-e90073b66501-kube-api-access-7xlmg\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:49.745460 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:49.745454    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8cda825-8d71-453b-9fe0-e90073b66501-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:40:50.095851 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.095818    2576 generic.go:356] "Generic (PLEG): container finished" podID="f8cda825-8d71-453b-9fe0-e90073b66501" containerID="2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8" exitCode=0
Sep 09 20:40:50.096036 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.095861    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"
Sep 09 20:40:50.096036 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.095878    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" event={"ID":"f8cda825-8d71-453b-9fe0-e90073b66501","Type":"ContainerDied","Data":"2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8"}
Sep 09 20:40:50.096036 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.095920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr" event={"ID":"f8cda825-8d71-453b-9fe0-e90073b66501","Type":"ContainerDied","Data":"b74c5ee25929ede688a56e8eb8e212c8639265b33f05961776d4aa8bb5b21198"}
Sep 09 20:40:50.096036 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.095941    2576 scope.go:117] "RemoveContainer" containerID="2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8"
Sep 09 20:40:50.104492 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.104472    2576 scope.go:117] "RemoveContainer" containerID="2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8"
Sep 09 20:40:50.104770 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:40:50.104752    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8\": container with ID starting with 2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8 not found: ID does not exist" containerID="2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8"
Sep 09 20:40:50.104846 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.104774    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8"} err="failed to get container status \"2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8\": rpc error: code = NotFound desc = could not find container \"2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8\": container with ID starting with 2898d0f31b149db49e2efc2ddfa84154058456e928e613f487503d409d7d9df8 not found: ID does not exist"
Sep 09 20:40:50.112673 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.112649    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"]
Sep 09 20:40:50.118178 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:50.118146    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvce55ae740357a3a31a27cdb8b66ffe20f-kserve-router-sche6zkhr"]
Sep 09 20:40:51.962283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:40:51.962253    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" path="/var/lib/kubelet/pods/f8cda825-8d71-453b-9fe0-e90073b66501/volumes"
Sep 09 20:41:03.132817 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.132775    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"]
Sep 09 20:41:03.133283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133219    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main"
Sep 09 20:41:03.133283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133239    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main"
Sep 09 20:41:03.133283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133262    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main"
Sep 09 20:41:03.133283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133271    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main"
Sep 09 20:41:03.133283 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133285    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" containerName="main"
Sep 09 20:41:03.133471 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133291    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" containerName="main"
Sep 09 20:41:03.133471 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133348    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8cda825-8d71-453b-9fe0-e90073b66501" containerName="main"
Sep 09 20:41:03.133471 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133355    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="101f1cab-35aa-4a5f-b831-bd25c777e9cd" containerName="main"
Sep 09 20:41:03.133471 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.133361    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9351fc59-169a-43d9-99cd-fb6fc7eebbd3" containerName="main"
Sep 09 20:41:03.138087 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.138055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.140517 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.140482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 09 20:41:03.141454 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.141434    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 09 20:41:03.141588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.141432    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 09 20:41:03.141588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.141476    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-9wq6g\""
Sep 09 20:41:03.147695 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.147673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"]
Sep 09 20:41:03.258308 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.258275    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.258308 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.258313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c87bee-66e9-4099-b5d0-aa353eacba0d-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.258553 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.258341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.258553 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.258396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.258553 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.258424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.258553 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.258472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbgk\" (UniqueName: \"kubernetes.io/projected/06c87bee-66e9-4099-b5d0-aa353eacba0d-kube-api-access-xjbgk\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c87bee-66e9-4099-b5d0-aa353eacba0d-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359714 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359714 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359488    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359714 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359714 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbgk\" (UniqueName: \"kubernetes.io/projected/06c87bee-66e9-4099-b5d0-aa353eacba0d-kube-api-access-xjbgk\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359945 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-home\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.359945 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-tmp-dir\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.360077 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.359994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-model-cache\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.361964 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.361934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-dshm\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.362366 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.362346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c87bee-66e9-4099-b5d0-aa353eacba0d-tls-certs\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.368177 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.368150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbgk\" (UniqueName: \"kubernetes.io/projected/06c87bee-66e9-4099-b5d0-aa353eacba0d-kube-api-access-xjbgk\") pod \"llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.449526 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.449428    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:03.730358 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:03.730211    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"]
Sep 09 20:41:03.733152 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:41:03.733123    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c87bee_66e9_4099_b5d0_aa353eacba0d.slice/crio-5620987a022a4ed53dd81424363b12bfa48f660f00ef2c618a0e51f70896c282 WatchSource:0}: Error finding container 5620987a022a4ed53dd81424363b12bfa48f660f00ef2c618a0e51f70896c282: Status 404 returned error can't find the container with id 5620987a022a4ed53dd81424363b12bfa48f660f00ef2c618a0e51f70896c282
Sep 09 20:41:04.166493 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:04.166454    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" event={"ID":"06c87bee-66e9-4099-b5d0-aa353eacba0d","Type":"ContainerStarted","Data":"c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"}
Sep 09 20:41:04.166909 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:04.166507    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" event={"ID":"06c87bee-66e9-4099-b5d0-aa353eacba0d","Type":"ContainerStarted","Data":"5620987a022a4ed53dd81424363b12bfa48f660f00ef2c618a0e51f70896c282"}
Sep 09 20:41:04.194923 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:04.194871    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" podStartSLOduration=1.194855168 podStartE2EDuration="1.194855168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:04.191502589 +0000 UTC m=+2766.709640116" watchObservedRunningTime="2026-09-09 20:41:04.194855168 +0000 UTC m=+2766.712992715"
Sep 09 20:41:04.450525 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:04.450440    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:13.450010 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:13.449971    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:13.462917 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:13.462887    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:14.228464 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:14.228428    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:41:43.590444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.590365    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"]
Sep 09 20:41:43.593281 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.593264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.598882 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.598847    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-4hzxg\""
Sep 09 20:41:43.599038 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.599014    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c3a07-f22f-4078-8947-91e47dda6fb7-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.599131 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.599080    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6w5m\" (UniqueName: \"kubernetes.io/projected/bc3c3a07-f22f-4078-8947-91e47dda6fb7-kube-api-access-d6w5m\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.600991 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.600505    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 09 20:41:43.609057 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.609032    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"]
Sep 09 20:41:43.700553 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.700521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c3a07-f22f-4078-8947-91e47dda6fb7-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.700730 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.700571    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6w5m\" (UniqueName: \"kubernetes.io/projected/bc3c3a07-f22f-4078-8947-91e47dda6fb7-kube-api-access-d6w5m\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.703192 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.703171    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c3a07-f22f-4078-8947-91e47dda6fb7-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.718566 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.718545    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6w5m\" (UniqueName: \"kubernetes.io/projected/bc3c3a07-f22f-4078-8947-91e47dda6fb7-kube-api-access-d6w5m\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:43.904909 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:43.904815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:44.180625 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:44.180599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"]
Sep 09 20:41:44.181934 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:41:44.181907    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3c3a07_f22f_4078_8947_91e47dda6fb7.slice/crio-7af99538fd2850eced3cddae66a2d03978972c65e50967b73615629f77c93bd9 WatchSource:0}: Error finding container 7af99538fd2850eced3cddae66a2d03978972c65e50967b73615629f77c93bd9: Status 404 returned error can't find the container with id 7af99538fd2850eced3cddae66a2d03978972c65e50967b73615629f77c93bd9
Sep 09 20:41:44.365107 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:44.365055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" event={"ID":"bc3c3a07-f22f-4078-8947-91e47dda6fb7","Type":"ContainerStarted","Data":"471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a"}
Sep 09 20:41:44.365308 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:44.365123    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" event={"ID":"bc3c3a07-f22f-4078-8947-91e47dda6fb7","Type":"ContainerStarted","Data":"7af99538fd2850eced3cddae66a2d03978972c65e50967b73615629f77c93bd9"}
Sep 09 20:41:44.365308 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:44.365165    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:41:44.383642 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:41:44.383592    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" podStartSLOduration=1.383575902 podStartE2EDuration="1.383575902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:44.381864576 +0000 UTC m=+2806.900002103" watchObservedRunningTime="2026-09-09 20:41:44.383575902 +0000 UTC m=+2806.901713429"
Sep 09 20:42:15.371913 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:15.371879    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:42:17.264444 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:17.264407    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"]
Sep 09 20:42:25.372216 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:25.372170    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:42:32.288514 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:32.288472    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main" containerID="cri-o://471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a" gracePeriod=45
Sep 09 20:42:35.372703 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:35.372661    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:42:45.372741 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:45.372697    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:42:45.373137 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:45.372810    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:42:55.371857 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:42:55.371819    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:43:02.541655 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.541624    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:43:02.640189 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.640143    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6w5m\" (UniqueName: \"kubernetes.io/projected/bc3c3a07-f22f-4078-8947-91e47dda6fb7-kube-api-access-d6w5m\") pod \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") "
Sep 09 20:43:02.640414 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.640231    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c3a07-f22f-4078-8947-91e47dda6fb7-tls-certs\") pod \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\" (UID: \"bc3c3a07-f22f-4078-8947-91e47dda6fb7\") "
Sep 09 20:43:02.642614 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.642586    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3c3a07-f22f-4078-8947-91e47dda6fb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc3c3a07-f22f-4078-8947-91e47dda6fb7" (UID: "bc3c3a07-f22f-4078-8947-91e47dda6fb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:02.642711 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.642587    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3c3a07-f22f-4078-8947-91e47dda6fb7-kube-api-access-d6w5m" (OuterVolumeSpecName: "kube-api-access-d6w5m") pod "bc3c3a07-f22f-4078-8947-91e47dda6fb7" (UID: "bc3c3a07-f22f-4078-8947-91e47dda6fb7"). InnerVolumeSpecName "kube-api-access-d6w5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:02.739394 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.739361    2576 generic.go:356] "Generic (PLEG): container finished" podID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerID="471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a" exitCode=0
Sep 09 20:43:02.739571 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.739408    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"
Sep 09 20:43:02.739571 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.739418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" event={"ID":"bc3c3a07-f22f-4078-8947-91e47dda6fb7","Type":"ContainerDied","Data":"471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a"}
Sep 09 20:43:02.739571 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.739460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg" event={"ID":"bc3c3a07-f22f-4078-8947-91e47dda6fb7","Type":"ContainerDied","Data":"7af99538fd2850eced3cddae66a2d03978972c65e50967b73615629f77c93bd9"}
Sep 09 20:43:02.739571 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.739482    2576 scope.go:117] "RemoveContainer" containerID="471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a"
Sep 09 20:43:02.741500 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.741475    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3c3a07-f22f-4078-8947-91e47dda6fb7-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:02.741602 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.741516    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6w5m\" (UniqueName: \"kubernetes.io/projected/bc3c3a07-f22f-4078-8947-91e47dda6fb7-kube-api-access-d6w5m\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:02.750876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.750856    2576 scope.go:117] "RemoveContainer" containerID="471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a"
Sep 09 20:43:02.751233 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:43:02.751210    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a\": container with ID starting with 471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a not found: ID does not exist" containerID="471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a"
Sep 09 20:43:02.751305 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.751243    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a"} err="failed to get container status \"471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a\": rpc error: code = NotFound desc = could not find container \"471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a\": container with ID starting with 471cd84dc8c86aafcba2c7106706965fd5d41d8cb88660649de2af8a324dbf8a not found: ID does not exist"
Sep 09 20:43:02.763115 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.763088    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"]
Sep 09 20:43:02.766324 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:02.766297    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-sche55zpg"]
Sep 09 20:43:03.962850 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:03.962815    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" path="/var/lib/kubelet/pods/bc3c3a07-f22f-4078-8947-91e47dda6fb7/volumes"
Sep 09 20:43:07.383364 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.383302    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"]
Sep 09 20:43:07.392686 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:43:07.392647    2576 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 09 20:43:07.392686 ip-10-0-135-242 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:43:07.392686 ip-10-0-135-242 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 20:43:07.392686 ip-10-0-135-242 kubenswrapper[2576]:  > containerID="c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea" cmd=["/bin/sleep","15"]
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.392698    2576 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" message="" err=<
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]:  >
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:43:07.392711    2576 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]:         , stdout: , stderr: , exit code -1
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]:  > pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" podUID="06c87bee-66e9-4099-b5d0-aa353eacba0d" containerName="main" containerID="cri-o://c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"
Sep 09 20:43:07.392949 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.392746    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" podUID="06c87bee-66e9-4099-b5d0-aa353eacba0d" containerName="main" containerID="cri-o://c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea" gracePeriod=60
Sep 09 20:43:07.753886 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.753860    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:43:07.766669 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.766585    2576 generic.go:356] "Generic (PLEG): container finished" podID="06c87bee-66e9-4099-b5d0-aa353eacba0d" containerID="c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea" exitCode=0
Sep 09 20:43:07.766669 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.766634    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"
Sep 09 20:43:07.766877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.766679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" event={"ID":"06c87bee-66e9-4099-b5d0-aa353eacba0d","Type":"ContainerDied","Data":"c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"}
Sep 09 20:43:07.766877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.766738    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7" event={"ID":"06c87bee-66e9-4099-b5d0-aa353eacba0d","Type":"ContainerDied","Data":"5620987a022a4ed53dd81424363b12bfa48f660f00ef2c618a0e51f70896c282"}
Sep 09 20:43:07.766877 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.766768    2576 scope.go:117] "RemoveContainer" containerID="c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"
Sep 09 20:43:07.776872 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.776846    2576 scope.go:117] "RemoveContainer" containerID="c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"
Sep 09 20:43:07.777262 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:43:07.777236    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea\": container with ID starting with c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea not found: ID does not exist" containerID="c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"
Sep 09 20:43:07.777375 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.777273    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea"} err="failed to get container status \"c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea\": rpc error: code = NotFound desc = could not find container \"c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea\": container with ID starting with c4efb959cdb75db42934c2beeb1e463d676b3a016bffbcb798c69694ea2affea not found: ID does not exist"
Sep 09 20:43:07.780579 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780558    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-home\") pod \"06c87bee-66e9-4099-b5d0-aa353eacba0d\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") "
Sep 09 20:43:07.780723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780610    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjbgk\" (UniqueName: \"kubernetes.io/projected/06c87bee-66e9-4099-b5d0-aa353eacba0d-kube-api-access-xjbgk\") pod \"06c87bee-66e9-4099-b5d0-aa353eacba0d\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") "
Sep 09 20:43:07.780723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780654    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-model-cache\") pod \"06c87bee-66e9-4099-b5d0-aa353eacba0d\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") "
Sep 09 20:43:07.780723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780671    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-dshm\") pod \"06c87bee-66e9-4099-b5d0-aa353eacba0d\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") "
Sep 09 20:43:07.780723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780688    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-tmp-dir\") pod \"06c87bee-66e9-4099-b5d0-aa353eacba0d\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") "
Sep 09 20:43:07.780723 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780721    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c87bee-66e9-4099-b5d0-aa353eacba0d-tls-certs\") pod \"06c87bee-66e9-4099-b5d0-aa353eacba0d\" (UID: \"06c87bee-66e9-4099-b5d0-aa353eacba0d\") "
Sep 09 20:43:07.781021 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780810    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-home" (OuterVolumeSpecName: "home") pod "06c87bee-66e9-4099-b5d0-aa353eacba0d" (UID: "06c87bee-66e9-4099-b5d0-aa353eacba0d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:07.781021 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.780906    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-model-cache" (OuterVolumeSpecName: "model-cache") pod "06c87bee-66e9-4099-b5d0-aa353eacba0d" (UID: "06c87bee-66e9-4099-b5d0-aa353eacba0d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:07.781021 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.781004    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-home\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:07.781184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.781022    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-model-cache\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:07.781242 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.781225    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06c87bee-66e9-4099-b5d0-aa353eacba0d" (UID: "06c87bee-66e9-4099-b5d0-aa353eacba0d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:07.783435 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.783399    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c87bee-66e9-4099-b5d0-aa353eacba0d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06c87bee-66e9-4099-b5d0-aa353eacba0d" (UID: "06c87bee-66e9-4099-b5d0-aa353eacba0d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:43:07.783582 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.783441    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c87bee-66e9-4099-b5d0-aa353eacba0d-kube-api-access-xjbgk" (OuterVolumeSpecName: "kube-api-access-xjbgk") pod "06c87bee-66e9-4099-b5d0-aa353eacba0d" (UID: "06c87bee-66e9-4099-b5d0-aa353eacba0d"). InnerVolumeSpecName "kube-api-access-xjbgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:43:07.783582 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.783474    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-dshm" (OuterVolumeSpecName: "dshm") pod "06c87bee-66e9-4099-b5d0-aa353eacba0d" (UID: "06c87bee-66e9-4099-b5d0-aa353eacba0d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:43:07.882064 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.882030    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjbgk\" (UniqueName: \"kubernetes.io/projected/06c87bee-66e9-4099-b5d0-aa353eacba0d-kube-api-access-xjbgk\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:07.882064 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.882058    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-dshm\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:07.882268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.882089    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c87bee-66e9-4099-b5d0-aa353eacba0d-tmp-dir\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:07.882268 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:07.882097    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c87bee-66e9-4099-b5d0-aa353eacba0d-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:43:08.082694 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:08.082660    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"]
Sep 09 20:43:08.089121 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:08.089093    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc-router-managed-test-llm-4b931143-kserve-85bf867969b56b7"]
Sep 09 20:43:09.962138 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:43:09.962103    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c87bee-66e9-4099-b5d0-aa353eacba0d" path="/var/lib/kubelet/pods/06c87bee-66e9-4099-b5d0-aa353eacba0d/volumes"
Sep 09 20:44:01.078876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.078842    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"]
Sep 09 20:44:01.079349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.079227    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main"
Sep 09 20:44:01.079349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.079245    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main"
Sep 09 20:44:01.079349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.079266    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c87bee-66e9-4099-b5d0-aa353eacba0d" containerName="main"
Sep 09 20:44:01.079349 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.079276    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c87bee-66e9-4099-b5d0-aa353eacba0d" containerName="main"
Sep 09 20:44:01.079600 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.079353    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3c3a07-f22f-4078-8947-91e47dda6fb7" containerName="main"
Sep 09 20:44:01.079600 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.079368    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c87bee-66e9-4099-b5d0-aa353eacba0d" containerName="main"
Sep 09 20:44:01.083937 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.083910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.087569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.087470    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisvc-model-fb-opt-125m-oci-r-0fba19de-epp-sa-dockercfg-cl29g\""
Sep 09 20:44:01.087569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.087484    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-vdznc\""
Sep 09 20:44:01.087569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.087498    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 09 20:44:01.087569 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.087517    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 09 20:44:01.087862 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.087588    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 09 20:44:01.094876 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.094846    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"]
Sep 09 20:44:01.214470 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.214435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e70b3c-d603-444a-b710-4c8b4a91f123-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.214616 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.214535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j67nw\" (UniqueName: \"kubernetes.io/projected/14e70b3c-d603-444a-b710-4c8b4a91f123-kube-api-access-j67nw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.315286 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.315253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e70b3c-d603-444a-b710-4c8b4a91f123-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.315456 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.315321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j67nw\" (UniqueName: \"kubernetes.io/projected/14e70b3c-d603-444a-b710-4c8b4a91f123-kube-api-access-j67nw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.317918 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.317899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e70b3c-d603-444a-b710-4c8b4a91f123-tls-certs\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.329630 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.329566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j67nw\" (UniqueName: \"kubernetes.io/projected/14e70b3c-d603-444a-b710-4c8b4a91f123-kube-api-access-j67nw\") pod \"llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.396276 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.396248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:01.667603 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:01.667564    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"]
Sep 09 20:44:01.669254 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:44:01.669217    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e70b3c_d603_444a_b710_4c8b4a91f123.slice/crio-c2b5577924fb57143c39b7a9a956b0fa710250d271ca1ca799eb0e38a23f2445 WatchSource:0}: Error finding container c2b5577924fb57143c39b7a9a956b0fa710250d271ca1ca799eb0e38a23f2445: Status 404 returned error can't find the container with id c2b5577924fb57143c39b7a9a956b0fa710250d271ca1ca799eb0e38a23f2445
Sep 09 20:44:02.022649 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:02.022557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" event={"ID":"14e70b3c-d603-444a-b710-4c8b4a91f123","Type":"ContainerStarted","Data":"6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e"}
Sep 09 20:44:02.022649 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:02.022613    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" event={"ID":"14e70b3c-d603-444a-b710-4c8b4a91f123","Type":"ContainerStarted","Data":"c2b5577924fb57143c39b7a9a956b0fa710250d271ca1ca799eb0e38a23f2445"}
Sep 09 20:44:02.022854 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:02.022691    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:02.039266 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:02.039206    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" podStartSLOduration=1.039187371 podStartE2EDuration="1.039187371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:02.038454256 +0000 UTC m=+2944.556591783" watchObservedRunningTime="2026-09-09 20:44:02.039187371 +0000 UTC m=+2944.557324899"
Sep 09 20:44:22.418940 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.418858    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"]
Sep 09 20:44:22.422495 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.422463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.427160 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.427137    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 09 20:44:22.427298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.427166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-6mcxv\""
Sep 09 20:44:22.427298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.427175    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 09 20:44:22.427298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.427136    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-ljlnf\""
Sep 09 20:44:22.427298 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.427136    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 09 20:44:22.450069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.450037    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"]
Sep 09 20:44:22.579320 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.579281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4lr\" (UniqueName: \"kubernetes.io/projected/2f619148-184c-42b6-893f-2b3ccc6fca2b-kube-api-access-5n4lr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.579320 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.579320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f619148-184c-42b6-893f-2b3ccc6fca2b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.679935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.679829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4lr\" (UniqueName: \"kubernetes.io/projected/2f619148-184c-42b6-893f-2b3ccc6fca2b-kube-api-access-5n4lr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.679935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.679875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f619148-184c-42b6-893f-2b3ccc6fca2b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.682709 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.682683    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f619148-184c-42b6-893f-2b3ccc6fca2b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.687607 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.687580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4lr\" (UniqueName: \"kubernetes.io/projected/2f619148-184c-42b6-893f-2b3ccc6fca2b-kube-api-access-5n4lr\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:22.733889 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:22.733854    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:23.008293 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:23.008207    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"]
Sep 09 20:44:23.129406 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:23.129302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" event={"ID":"2f619148-184c-42b6-893f-2b3ccc6fca2b","Type":"ContainerStarted","Data":"22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627"}
Sep 09 20:44:23.129406 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:23.129370    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" event={"ID":"2f619148-184c-42b6-893f-2b3ccc6fca2b","Type":"ContainerStarted","Data":"7cb2ba7a10f66943964fa50705ed478525224a8a0c0b0427f396151aeaf5361d"}
Sep 09 20:44:23.129680 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:23.129454    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:23.146561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:23.146501    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podStartSLOduration=1.146485616 podStartE2EDuration="1.146485616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:44:23.145345622 +0000 UTC m=+2965.663483141" watchObservedRunningTime="2026-09-09 20:44:23.146485616 +0000 UTC m=+2965.664623178"
Sep 09 20:44:33.029663 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:33.029633    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:44:54.136342 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:54.136316    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:44:58.176528 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:58.176502    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:44:58.185932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:44:58.185895    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:46:17.103506 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:17.103470    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"]
Sep 09 20:46:21.593894 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:21.593860    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"]
Sep 09 20:46:23.032350 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:23.032299    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:24.136426 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:24.136391    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:32.123563 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:32.123522    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main" containerID="cri-o://6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e" gracePeriod=45
Sep 09 20:46:33.030043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:33.030007    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:34.136611 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:34.136568    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:36.615436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:36.615390    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main" containerID="cri-o://22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627" gracePeriod=45
Sep 09 20:46:43.030167 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:43.030128    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:43.030561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:43.030248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:46:44.135754 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:44.135721    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:44.136194 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:44.135818    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:46:53.029822 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:53.029778    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:46:54.136205 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:46:54.136171    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:47:02.374159 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.374132    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:47:02.454029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.453916    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j67nw\" (UniqueName: \"kubernetes.io/projected/14e70b3c-d603-444a-b710-4c8b4a91f123-kube-api-access-j67nw\") pod \"14e70b3c-d603-444a-b710-4c8b4a91f123\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") "
Sep 09 20:47:02.454029 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.454005    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e70b3c-d603-444a-b710-4c8b4a91f123-tls-certs\") pod \"14e70b3c-d603-444a-b710-4c8b4a91f123\" (UID: \"14e70b3c-d603-444a-b710-4c8b4a91f123\") "
Sep 09 20:47:02.456382 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.456355    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e70b3c-d603-444a-b710-4c8b4a91f123-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14e70b3c-d603-444a-b710-4c8b4a91f123" (UID: "14e70b3c-d603-444a-b710-4c8b4a91f123"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:02.456474 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.456429    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e70b3c-d603-444a-b710-4c8b4a91f123-kube-api-access-j67nw" (OuterVolumeSpecName: "kube-api-access-j67nw") pod "14e70b3c-d603-444a-b710-4c8b4a91f123" (UID: "14e70b3c-d603-444a-b710-4c8b4a91f123"). InnerVolumeSpecName "kube-api-access-j67nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:02.554752 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.554706    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j67nw\" (UniqueName: \"kubernetes.io/projected/14e70b3c-d603-444a-b710-4c8b4a91f123-kube-api-access-j67nw\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:47:02.554752 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.554742    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e70b3c-d603-444a-b710-4c8b4a91f123-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:47:02.914743 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.914709    2576 generic.go:356] "Generic (PLEG): container finished" podID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerID="6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e" exitCode=0
Sep 09 20:47:02.914939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.914753    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"
Sep 09 20:47:02.914939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.914802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" event={"ID":"14e70b3c-d603-444a-b710-4c8b4a91f123","Type":"ContainerDied","Data":"6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e"}
Sep 09 20:47:02.914939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.914864    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc" event={"ID":"14e70b3c-d603-444a-b710-4c8b4a91f123","Type":"ContainerDied","Data":"c2b5577924fb57143c39b7a9a956b0fa710250d271ca1ca799eb0e38a23f2445"}
Sep 09 20:47:02.914939 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.914893    2576 scope.go:117] "RemoveContainer" containerID="6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e"
Sep 09 20:47:02.924019 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.924002    2576 scope.go:117] "RemoveContainer" containerID="6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e"
Sep 09 20:47:02.924302 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:47:02.924283    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e\": container with ID starting with 6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e not found: ID does not exist" containerID="6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e"
Sep 09 20:47:02.924366 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.924314    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e"} err="failed to get container status \"6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e\": rpc error: code = NotFound desc = could not find container \"6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e\": container with ID starting with 6835338435bad46451ce21167f31974ba3f23d4be90ec083f69c519a253c905e not found: ID does not exist"
Sep 09 20:47:02.960334 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.960305    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"]
Sep 09 20:47:02.973584 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:02.973551    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc20d1e8ef181cac7effb0fab7b8590b15-kserve-router-scheh9ptc"]
Sep 09 20:47:03.961628 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:03.961591    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" path="/var/lib/kubelet/pods/14e70b3c-d603-444a-b710-4c8b4a91f123/volumes"
Sep 09 20:47:04.136441 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:04.136398    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:47:06.854145 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.854121    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:47:06.936031 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.935929    2576 generic.go:356] "Generic (PLEG): container finished" podID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerID="22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627" exitCode=0
Sep 09 20:47:06.936031 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.936000    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"
Sep 09 20:47:06.936031 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.936020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" event={"ID":"2f619148-184c-42b6-893f-2b3ccc6fca2b","Type":"ContainerDied","Data":"22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627"}
Sep 09 20:47:06.936272 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.936086    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj" event={"ID":"2f619148-184c-42b6-893f-2b3ccc6fca2b","Type":"ContainerDied","Data":"7cb2ba7a10f66943964fa50705ed478525224a8a0c0b0427f396151aeaf5361d"}
Sep 09 20:47:06.936272 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.936104    2576 scope.go:117] "RemoveContainer" containerID="22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627"
Sep 09 20:47:06.944355 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.944338    2576 scope.go:117] "RemoveContainer" containerID="22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627"
Sep 09 20:47:06.944638 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:47:06.944620    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627\": container with ID starting with 22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627 not found: ID does not exist" containerID="22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627"
Sep 09 20:47:06.944691 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.944646    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627"} err="failed to get container status \"22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627\": rpc error: code = NotFound desc = could not find container \"22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627\": container with ID starting with 22854270ad103622dfa9feabbc326708ab9754a94567b1e3bdcb6045211f5627 not found: ID does not exist"
Sep 09 20:47:06.991896 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.991866    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n4lr\" (UniqueName: \"kubernetes.io/projected/2f619148-184c-42b6-893f-2b3ccc6fca2b-kube-api-access-5n4lr\") pod \"2f619148-184c-42b6-893f-2b3ccc6fca2b\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") "
Sep 09 20:47:06.992061 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.991919    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f619148-184c-42b6-893f-2b3ccc6fca2b-tls-certs\") pod \"2f619148-184c-42b6-893f-2b3ccc6fca2b\" (UID: \"2f619148-184c-42b6-893f-2b3ccc6fca2b\") "
Sep 09 20:47:06.994362 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.994315    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f619148-184c-42b6-893f-2b3ccc6fca2b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f619148-184c-42b6-893f-2b3ccc6fca2b" (UID: "2f619148-184c-42b6-893f-2b3ccc6fca2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:06.994577 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:06.994549    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f619148-184c-42b6-893f-2b3ccc6fca2b-kube-api-access-5n4lr" (OuterVolumeSpecName: "kube-api-access-5n4lr") pod "2f619148-184c-42b6-893f-2b3ccc6fca2b" (UID: "2f619148-184c-42b6-893f-2b3ccc6fca2b"). InnerVolumeSpecName "kube-api-access-5n4lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:07.093127 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:07.093091    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f619148-184c-42b6-893f-2b3ccc6fca2b-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:47:07.093127 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:07.093122    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n4lr\" (UniqueName: \"kubernetes.io/projected/2f619148-184c-42b6-893f-2b3ccc6fca2b-kube-api-access-5n4lr\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:47:07.298354 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:07.298321    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"]
Sep 09 20:47:07.307846 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:07.307816    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schenc6pj"]
Sep 09 20:47:07.961445 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:07.961409    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" path="/var/lib/kubelet/pods/2f619148-184c-42b6-893f-2b3ccc6fca2b/volumes"
Sep 09 20:47:15.577002 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.576857    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"]
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.577274    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.577288    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.577304    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.577310    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.577358    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e70b3c-d603-444a-b710-4c8b4a91f123" containerName="main"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.577366    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f619148-184c-42b6-893f-2b3ccc6fca2b" containerName="main"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.580137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.586117    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"kube-root-ca.crt\""
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.586132    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"openshift-service-ca.crt\""
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.586120    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-epp-sa-dockercfg-fmjsj\""
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.586117    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Sep 09 20:47:15.594871 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.586204    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-caba0b4a\"/\"default-dockercfg-4k47c\""
Sep 09 20:47:15.606274 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.606247    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"]
Sep 09 20:47:15.668192 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.668159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254027-4730-4e1b-b34f-0f3e5d4f4705-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.668409 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.668206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk87\" (UniqueName: \"kubernetes.io/projected/dd254027-4730-4e1b-b34f-0f3e5d4f4705-kube-api-access-kpk87\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.769590 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.769554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254027-4730-4e1b-b34f-0f3e5d4f4705-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.769813 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.769622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk87\" (UniqueName: \"kubernetes.io/projected/dd254027-4730-4e1b-b34f-0f3e5d4f4705-kube-api-access-kpk87\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.772375 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.772353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254027-4730-4e1b-b34f-0f3e5d4f4705-tls-certs\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.787158 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.787125    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk87\" (UniqueName: \"kubernetes.io/projected/dd254027-4730-4e1b-b34f-0f3e5d4f4705-kube-api-access-kpk87\") pod \"custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") " pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:15.890651 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:15.890612    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:16.216436 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:16.216407    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"]
Sep 09 20:47:16.217546 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:47:16.217520    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd254027_4730_4e1b_b34f_0f3e5d4f4705.slice/crio-804da4b453530bd2fab4b9ba6189aa561deff6f4a622c6ff63edaadb8b9d5890 WatchSource:0}: Error finding container 804da4b453530bd2fab4b9ba6189aa561deff6f4a622c6ff63edaadb8b9d5890: Status 404 returned error can't find the container with id 804da4b453530bd2fab4b9ba6189aa561deff6f4a622c6ff63edaadb8b9d5890
Sep 09 20:47:16.988368 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:16.988332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" event={"ID":"dd254027-4730-4e1b-b34f-0f3e5d4f4705","Type":"ContainerStarted","Data":"f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16"}
Sep 09 20:47:16.988785 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:16.988389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" event={"ID":"dd254027-4730-4e1b-b34f-0f3e5d4f4705","Type":"ContainerStarted","Data":"804da4b453530bd2fab4b9ba6189aa561deff6f4a622c6ff63edaadb8b9d5890"}
Sep 09 20:47:16.988785 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:16.988418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:47:17.025761 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:17.025709    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" podStartSLOduration=2.02569182 podStartE2EDuration="2.02569182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:17.018167241 +0000 UTC m=+3139.536304768" watchObservedRunningTime="2026-09-09 20:47:17.02569182 +0000 UTC m=+3139.543829346"
Sep 09 20:47:47.995271 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:47:47.995189    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:49:30.826120 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:30.826082    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"]
Sep 09 20:49:37.994848 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:37.994813    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:45.846586 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:45.846538    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main" containerID="cri-o://f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16" gracePeriod=45
Sep 09 20:49:47.994744 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:47.994701    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:57.995117 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:57.995076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:49:57.995535 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:57.995185    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:49:58.208503 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:58.208472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:49:58.217215 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:49:58.217192    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:50:07.995524 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:07.995486    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:50:16.091513 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.091482    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:50:16.153474 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.153390    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254027-4730-4e1b-b34f-0f3e5d4f4705-tls-certs\") pod \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") "
Sep 09 20:50:16.153474 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.153437    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpk87\" (UniqueName: \"kubernetes.io/projected/dd254027-4730-4e1b-b34f-0f3e5d4f4705-kube-api-access-kpk87\") pod \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\" (UID: \"dd254027-4730-4e1b-b34f-0f3e5d4f4705\") "
Sep 09 20:50:16.155751 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.155717    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd254027-4730-4e1b-b34f-0f3e5d4f4705-kube-api-access-kpk87" (OuterVolumeSpecName: "kube-api-access-kpk87") pod "dd254027-4730-4e1b-b34f-0f3e5d4f4705" (UID: "dd254027-4730-4e1b-b34f-0f3e5d4f4705"). InnerVolumeSpecName "kube-api-access-kpk87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:16.155868 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.155750    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd254027-4730-4e1b-b34f-0f3e5d4f4705-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dd254027-4730-4e1b-b34f-0f3e5d4f4705" (UID: "dd254027-4730-4e1b-b34f-0f3e5d4f4705"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:16.254670 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.254637    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd254027-4730-4e1b-b34f-0f3e5d4f4705-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:50:16.254670 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.254662    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpk87\" (UniqueName: \"kubernetes.io/projected/dd254027-4730-4e1b-b34f-0f3e5d4f4705-kube-api-access-kpk87\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:50:16.846525 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.846492    2576 generic.go:356] "Generic (PLEG): container finished" podID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerID="f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16" exitCode=0
Sep 09 20:50:16.846732 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.846538    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"
Sep 09 20:50:16.846732 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.846591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" event={"ID":"dd254027-4730-4e1b-b34f-0f3e5d4f4705","Type":"ContainerDied","Data":"f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16"}
Sep 09 20:50:16.846732 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.846653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5" event={"ID":"dd254027-4730-4e1b-b34f-0f3e5d4f4705","Type":"ContainerDied","Data":"804da4b453530bd2fab4b9ba6189aa561deff6f4a622c6ff63edaadb8b9d5890"}
Sep 09 20:50:16.846732 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.846683    2576 scope.go:117] "RemoveContainer" containerID="f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16"
Sep 09 20:50:16.855616 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.855599    2576 scope.go:117] "RemoveContainer" containerID="f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16"
Sep 09 20:50:16.855856 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:50:16.855834    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16\": container with ID starting with f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16 not found: ID does not exist" containerID="f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16"
Sep 09 20:50:16.855916 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.855867    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16"} err="failed to get container status \"f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16\": rpc error: code = NotFound desc = could not find container \"f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16\": container with ID starting with f0cdaf09ed643830e82dbdc95f30498aab788f37d03688858dd039e4bf8d9b16 not found: ID does not exist"
Sep 09 20:50:16.870299 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.870270    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"]
Sep 09 20:50:16.874666 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:16.874647    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-caba0b4a/custom-route-timeout-test-kserve-router-scheduler-54cbcc589ktj5"]
Sep 09 20:50:17.961967 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:17.961932    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" path="/var/lib/kubelet/pods/dd254027-4730-4e1b-b34f-0f3e5d4f4705/volumes"
Sep 09 20:50:34.942537 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.942498    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"]
Sep 09 20:50:34.942942 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.942829    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main"
Sep 09 20:50:34.942942 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.942842    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main"
Sep 09 20:50:34.942942 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.942907    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd254027-4730-4e1b-b34f-0f3e5d4f4705" containerName="main"
Sep 09 20:50:34.947494 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.947470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:34.950135 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.950110    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-kserve-self-signed-certs\""
Sep 09 20:50:34.951161 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.951140    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"default-dockercfg-k2t9w\""
Sep 09 20:50:34.951288 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.951189    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"kube-root-ca.crt\""
Sep 09 20:50:34.951288 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.951256    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"openshift-service-ca.crt\""
Sep 09 20:50:34.951411 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.951319    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b\"/\"e2e-wk-cfg-implicit-svc-epp-sa-dockercfg-cq26c\""
Sep 09 20:50:34.958614 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:34.958591    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"]
Sep 09 20:50:35.118792 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.118755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgjx4\" (UniqueName: \"kubernetes.io/projected/9a800515-268c-425b-8f02-5527e25b990e-kube-api-access-lgjx4\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.119016 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.118900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a800515-268c-425b-8f02-5527e25b990e-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.220047 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.219923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgjx4\" (UniqueName: \"kubernetes.io/projected/9a800515-268c-425b-8f02-5527e25b990e-kube-api-access-lgjx4\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.220047 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.220038    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a800515-268c-425b-8f02-5527e25b990e-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.222743 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.222710    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a800515-268c-425b-8f02-5527e25b990e-tls-certs\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.229471 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.229450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgjx4\" (UniqueName: \"kubernetes.io/projected/9a800515-268c-425b-8f02-5527e25b990e-kube-api-access-lgjx4\") pod \"e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") " pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.259158 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.259127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.533745 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.533720    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"]
Sep 09 20:50:35.535613 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:50:35.535583    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a800515_268c_425b_8f02_5527e25b990e.slice/crio-a8e02d7e61594945e41b7cf3932049d54ac7eb1c8ab9f6b15b4bd09694b98561 WatchSource:0}: Error finding container a8e02d7e61594945e41b7cf3932049d54ac7eb1c8ab9f6b15b4bd09694b98561: Status 404 returned error can't find the container with id a8e02d7e61594945e41b7cf3932049d54ac7eb1c8ab9f6b15b4bd09694b98561
Sep 09 20:50:35.944295 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.944259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" event={"ID":"9a800515-268c-425b-8f02-5527e25b990e","Type":"ContainerStarted","Data":"d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d"}
Sep 09 20:50:35.944726 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.944308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" event={"ID":"9a800515-268c-425b-8f02-5527e25b990e","Type":"ContainerStarted","Data":"a8e02d7e61594945e41b7cf3932049d54ac7eb1c8ab9f6b15b4bd09694b98561"}
Sep 09 20:50:35.944726 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.944396    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:50:35.962684 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:35.962632    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" podStartSLOduration=1.96261796 podStartE2EDuration="1.96261796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:35.96042807 +0000 UTC m=+3338.478565597" watchObservedRunningTime="2026-09-09 20:50:35.96261796 +0000 UTC m=+3338.480755508"
Sep 09 20:50:49.555043 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.555006    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"]
Sep 09 20:50:49.558217 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.558200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.562944 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.562853    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 09 20:50:49.562944 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.562880    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 09 20:50:49.562944 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.562860    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-sjljx\""
Sep 09 20:50:49.562944 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.562933    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 09 20:50:49.563266 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.562860    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-8x4sb\""
Sep 09 20:50:49.572805 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.572782    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"]
Sep 09 20:50:49.635982 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.635913    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.636195 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.636041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d586l\" (UniqueName: \"kubernetes.io/projected/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-kube-api-access-d586l\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.736745 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.736708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.736932 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.736774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d586l\" (UniqueName: \"kubernetes.io/projected/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-kube-api-access-d586l\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.739474 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.739447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.748281 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.748261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d586l\" (UniqueName: \"kubernetes.io/projected/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-kube-api-access-d586l\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:49.870940 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:49.870903    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:50.145822 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:50.145786    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"]
Sep 09 20:50:50.148426 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:50:50.148400    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9644e3bb_ecbf_4f10_b07e_ccca5e6442ef.slice/crio-b9d8d4c79235fc1f992f0bb831d9da5dfa9abc232448d43b10d94f21b310d615 WatchSource:0}: Error finding container b9d8d4c79235fc1f992f0bb831d9da5dfa9abc232448d43b10d94f21b310d615: Status 404 returned error can't find the container with id b9d8d4c79235fc1f992f0bb831d9da5dfa9abc232448d43b10d94f21b310d615
Sep 09 20:50:51.013781 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:51.013735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" event={"ID":"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef","Type":"ContainerStarted","Data":"a541d0966730243ee451cdfff36d876d0353014210ed549f900a6159557722e2"}
Sep 09 20:50:51.014266 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:51.013795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" event={"ID":"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef","Type":"ContainerStarted","Data":"b9d8d4c79235fc1f992f0bb831d9da5dfa9abc232448d43b10d94f21b310d615"}
Sep 09 20:50:51.014266 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:51.013823    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:50:51.038248 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:51.038151    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" podStartSLOduration=2.038137287 podStartE2EDuration="2.038137287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:51.036275094 +0000 UTC m=+3353.554412618" watchObservedRunningTime="2026-09-09 20:50:51.038137287 +0000 UTC m=+3353.556274813"
Sep 09 20:50:51.279133 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:50:51.279053    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"]
Sep 09 20:51:06.300810 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:06.300766    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main" containerID="cri-o://d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d" gracePeriod=45
Sep 09 20:51:11.281669 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:11.281629    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:51:21.281277 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:21.281242    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:51:22.020346 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:22.020317    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:51:31.281561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:31.281510    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:51:36.552506 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.552464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:51:36.731475 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.731436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a800515-268c-425b-8f02-5527e25b990e-tls-certs\") pod \"9a800515-268c-425b-8f02-5527e25b990e\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") "
Sep 09 20:51:36.731685 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.731490    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgjx4\" (UniqueName: \"kubernetes.io/projected/9a800515-268c-425b-8f02-5527e25b990e-kube-api-access-lgjx4\") pod \"9a800515-268c-425b-8f02-5527e25b990e\" (UID: \"9a800515-268c-425b-8f02-5527e25b990e\") "
Sep 09 20:51:36.733992 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.733942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a800515-268c-425b-8f02-5527e25b990e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a800515-268c-425b-8f02-5527e25b990e" (UID: "9a800515-268c-425b-8f02-5527e25b990e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:51:36.733992 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.733974    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a800515-268c-425b-8f02-5527e25b990e-kube-api-access-lgjx4" (OuterVolumeSpecName: "kube-api-access-lgjx4") pod "9a800515-268c-425b-8f02-5527e25b990e" (UID: "9a800515-268c-425b-8f02-5527e25b990e"). InnerVolumeSpecName "kube-api-access-lgjx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:51:36.832193 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.832160    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a800515-268c-425b-8f02-5527e25b990e-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:51:36.832193 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:36.832188    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgjx4\" (UniqueName: \"kubernetes.io/projected/9a800515-268c-425b-8f02-5527e25b990e-kube-api-access-lgjx4\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:51:37.240859 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.240820    2576 generic.go:356] "Generic (PLEG): container finished" podID="9a800515-268c-425b-8f02-5527e25b990e" containerID="d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d" exitCode=0
Sep 09 20:51:37.240859 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.240866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" event={"ID":"9a800515-268c-425b-8f02-5527e25b990e","Type":"ContainerDied","Data":"d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d"}
Sep 09 20:51:37.241160 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.240903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz" event={"ID":"9a800515-268c-425b-8f02-5527e25b990e","Type":"ContainerDied","Data":"a8e02d7e61594945e41b7cf3932049d54ac7eb1c8ab9f6b15b4bd09694b98561"}
Sep 09 20:51:37.241160 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.240924    2576 scope.go:117] "RemoveContainer" containerID="d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d"
Sep 09 20:51:37.241160 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.240867    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"
Sep 09 20:51:37.250199 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.250180    2576 scope.go:117] "RemoveContainer" containerID="d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d"
Sep 09 20:51:37.250465 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:51:37.250446    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d\": container with ID starting with d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d not found: ID does not exist" containerID="d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d"
Sep 09 20:51:37.250531 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.250480    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d"} err="failed to get container status \"d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d\": rpc error: code = NotFound desc = could not find container \"d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d\": container with ID starting with d8f9dc279a369cbe87c8213121fa2389176e8e3ad44f89e3eccaff7d545ec14d not found: ID does not exist"
Sep 09 20:51:37.263986 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.263941    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"]
Sep 09 20:51:37.268935 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.268902    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-well-known-config-deletion-blocked-by-implici-6b767e7b/e2e-wk-cfg-implicit-svc-kserve-router-scheduler-7887448cddj65kz"]
Sep 09 20:51:37.962297 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:51:37.962264    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a800515-268c-425b-8f02-5527e25b990e" path="/var/lib/kubelet/pods/9a800515-268c-425b-8f02-5527e25b990e/volumes"
Sep 09 20:53:04.700274 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:04.700224    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"]
Sep 09 20:53:12.020451 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:12.020407    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:53:19.722004 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:19.721898    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main" containerID="cri-o://a541d0966730243ee451cdfff36d876d0353014210ed549f900a6159557722e2" gracePeriod=45
Sep 09 20:53:22.020929 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:22.020889    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:53:32.020925 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:32.020882    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:53:32.021402 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:32.021029    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:53:42.021069 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.021020    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:53:42.287403 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.287322    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"]
Sep 09 20:53:42.287647 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.287636    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main"
Sep 09 20:53:42.287690 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.287649    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main"
Sep 09 20:53:42.287725 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.287713    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a800515-268c-425b-8f02-5527e25b990e" containerName="main"
Sep 09 20:53:42.290611 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.290592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.295659 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.295627    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 09 20:53:42.296624 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.296600    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-6htds\""
Sep 09 20:53:42.296810 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.296787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 09 20:53:42.296971 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.296934    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 09 20:53:42.297364 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.297100    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-6xks6\""
Sep 09 20:53:42.324239 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.324214    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"]
Sep 09 20:53:42.435188 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.435155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d7c80-b1ae-42ed-a89c-424b764b284f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.435353 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.435215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l24n\" (UniqueName: \"kubernetes.io/projected/745d7c80-b1ae-42ed-a89c-424b764b284f-kube-api-access-2l24n\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.535821 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.535777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l24n\" (UniqueName: \"kubernetes.io/projected/745d7c80-b1ae-42ed-a89c-424b764b284f-kube-api-access-2l24n\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.536040 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.535859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d7c80-b1ae-42ed-a89c-424b764b284f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.538536 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.538470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d7c80-b1ae-42ed-a89c-424b764b284f-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.560691 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.560658    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l24n\" (UniqueName: \"kubernetes.io/projected/745d7c80-b1ae-42ed-a89c-424b764b284f-kube-api-access-2l24n\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.600340 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.600301    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:42.894616 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:42.894585    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"]
Sep 09 20:53:42.897745 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:53:42.897719    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745d7c80_b1ae_42ed_a89c_424b764b284f.slice/crio-1287fbfcfded7142cd662bfa14df4ef7afa5a32ed148eaff6796e4238520f94c WatchSource:0}: Error finding container 1287fbfcfded7142cd662bfa14df4ef7afa5a32ed148eaff6796e4238520f94c: Status 404 returned error can't find the container with id 1287fbfcfded7142cd662bfa14df4ef7afa5a32ed148eaff6796e4238520f94c
Sep 09 20:53:43.838542 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:43.838503    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" event={"ID":"745d7c80-b1ae-42ed-a89c-424b764b284f","Type":"ContainerStarted","Data":"2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d"}
Sep 09 20:53:43.839021 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:43.838562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" event={"ID":"745d7c80-b1ae-42ed-a89c-424b764b284f","Type":"ContainerStarted","Data":"1287fbfcfded7142cd662bfa14df4ef7afa5a32ed148eaff6796e4238520f94c"}
Sep 09 20:53:43.839021 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:43.838592    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:53:43.877895 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:43.877840    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podStartSLOduration=1.877823907 podStartE2EDuration="1.877823907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:43.877471306 +0000 UTC m=+3526.395608856" watchObservedRunningTime="2026-09-09 20:53:43.877823907 +0000 UTC m=+3526.395961434"
Sep 09 20:53:49.869993 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:49.869932    2576 generic.go:356] "Generic (PLEG): container finished" podID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerID="a541d0966730243ee451cdfff36d876d0353014210ed549f900a6159557722e2" exitCode=0
Sep 09 20:53:49.870360 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:49.869992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" event={"ID":"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef","Type":"ContainerDied","Data":"a541d0966730243ee451cdfff36d876d0353014210ed549f900a6159557722e2"}
Sep 09 20:53:49.974785 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:49.974762    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:53:50.101321 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.101290    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-tls-certs\") pod \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") "
Sep 09 20:53:50.101485 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.101355    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d586l\" (UniqueName: \"kubernetes.io/projected/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-kube-api-access-d586l\") pod \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\" (UID: \"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef\") "
Sep 09 20:53:50.104286 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.104258    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" (UID: "9644e3bb-ecbf-4f10-b07e-ccca5e6442ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:53:50.104286 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.104270    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-kube-api-access-d586l" (OuterVolumeSpecName: "kube-api-access-d586l") pod "9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" (UID: "9644e3bb-ecbf-4f10-b07e-ccca5e6442ef"). InnerVolumeSpecName "kube-api-access-d586l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:53:50.203149 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.203113    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d586l\" (UniqueName: \"kubernetes.io/projected/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-kube-api-access-d586l\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.203149 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.203146    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:53:50.876056 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.876027    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"
Sep 09 20:53:50.876561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.876064    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm" event={"ID":"9644e3bb-ecbf-4f10-b07e-ccca5e6442ef","Type":"ContainerDied","Data":"b9d8d4c79235fc1f992f0bb831d9da5dfa9abc232448d43b10d94f21b310d615"}
Sep 09 20:53:50.876561 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.876134    2576 scope.go:117] "RemoveContainer" containerID="a541d0966730243ee451cdfff36d876d0353014210ed549f900a6159557722e2"
Sep 09 20:53:50.911975 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.911924    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"]
Sep 09 20:53:50.913705 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:50.913683    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-ggbgm"]
Sep 09 20:53:51.961777 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:53:51.961742    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" path="/var/lib/kubelet/pods/9644e3bb-ecbf-4f10-b07e-ccca5e6442ef/volumes"
Sep 09 20:54:12.413825 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.413787    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"]
Sep 09 20:54:12.414256 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.414160    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main"
Sep 09 20:54:12.414256 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.414174    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main"
Sep 09 20:54:12.414256 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.414228    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9644e3bb-ecbf-4f10-b07e-ccca5e6442ef" containerName="main"
Sep 09 20:54:12.417148 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.417127    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.421554 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.421530    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-7qhvs\""
Sep 09 20:54:12.421693 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.421601    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 09 20:54:12.421693 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.421628    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 09 20:54:12.421693 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.421679    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 09 20:54:12.422037 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.422020    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-zrrxw\""
Sep 09 20:54:12.428899 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.428874    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"]
Sep 09 20:54:12.480921 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.480879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fdf39ca-9420-4c22-b111-3567ad1d5869-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.481123 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.480935    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n49s\" (UniqueName: \"kubernetes.io/projected/9fdf39ca-9420-4c22-b111-3567ad1d5869-kube-api-access-5n49s\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.581581 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.581545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n49s\" (UniqueName: \"kubernetes.io/projected/9fdf39ca-9420-4c22-b111-3567ad1d5869-kube-api-access-5n49s\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.581763 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.581625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fdf39ca-9420-4c22-b111-3567ad1d5869-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.584265 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.584240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fdf39ca-9420-4c22-b111-3567ad1d5869-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.594759 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.594725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n49s\" (UniqueName: \"kubernetes.io/projected/9fdf39ca-9420-4c22-b111-3567ad1d5869-kube-api-access-5n49s\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:12.728330 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:12.728233    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:13.005177 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:13.004897    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"]
Sep 09 20:54:13.008028 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:54:13.007995    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdf39ca_9420_4c22_b111_3567ad1d5869.slice/crio-8074ca6a331779fd7890d2473cafa37c66fca398286f597072309f5429532b7c WatchSource:0}: Error finding container 8074ca6a331779fd7890d2473cafa37c66fca398286f597072309f5429532b7c: Status 404 returned error can't find the container with id 8074ca6a331779fd7890d2473cafa37c66fca398286f597072309f5429532b7c
Sep 09 20:54:13.992213 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:13.992177    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" event={"ID":"9fdf39ca-9420-4c22-b111-3567ad1d5869","Type":"ContainerStarted","Data":"2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7"}
Sep 09 20:54:13.992660 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:13.992240    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" event={"ID":"9fdf39ca-9420-4c22-b111-3567ad1d5869","Type":"ContainerStarted","Data":"8074ca6a331779fd7890d2473cafa37c66fca398286f597072309f5429532b7c"}
Sep 09 20:54:13.992660 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:13.992267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:14.012243 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:14.012177    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podStartSLOduration=2.012162284 podStartE2EDuration="2.012162284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:14.009677063 +0000 UTC m=+3556.527814589" watchObservedRunningTime="2026-09-09 20:54:14.012162284 +0000 UTC m=+3556.530299811"
Sep 09 20:54:14.844642 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:14.844615    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:54:44.999761 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:44.999725    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:54:58.239601 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:58.239575    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:54:58.249304 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:54:58.249282    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:55:53.857511 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:55:53.857477    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"]
Sep 09 20:55:54.845309 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:55:54.845267    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:04.845403 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:04.845361    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:08.879223 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:08.879184    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main" containerID="cri-o://2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d" gracePeriod=45
Sep 09 20:56:14.844536 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:14.844488    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:14.845149 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:14.844609    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:56:24.845601 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:24.845514    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:32.419452 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:32.419417    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"]
Sep 09 20:56:34.845139 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:34.845100    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:34.999788 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:34.999752    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:39.144105 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.144041    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:56:39.194543 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.194508    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d7c80-b1ae-42ed-a89c-424b764b284f-tls-certs\") pod \"745d7c80-b1ae-42ed-a89c-424b764b284f\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") "
Sep 09 20:56:39.194729 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.194558    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2l24n\" (UniqueName: \"kubernetes.io/projected/745d7c80-b1ae-42ed-a89c-424b764b284f-kube-api-access-2l24n\") pod \"745d7c80-b1ae-42ed-a89c-424b764b284f\" (UID: \"745d7c80-b1ae-42ed-a89c-424b764b284f\") "
Sep 09 20:56:39.196928 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.196900    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/745d7c80-b1ae-42ed-a89c-424b764b284f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "745d7c80-b1ae-42ed-a89c-424b764b284f" (UID: "745d7c80-b1ae-42ed-a89c-424b764b284f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:39.197107 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.196932    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745d7c80-b1ae-42ed-a89c-424b764b284f-kube-api-access-2l24n" (OuterVolumeSpecName: "kube-api-access-2l24n") pod "745d7c80-b1ae-42ed-a89c-424b764b284f" (UID: "745d7c80-b1ae-42ed-a89c-424b764b284f"). InnerVolumeSpecName "kube-api-access-2l24n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:39.295629 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.295590    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/745d7c80-b1ae-42ed-a89c-424b764b284f-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:56:39.295629 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.295624    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l24n\" (UniqueName: \"kubernetes.io/projected/745d7c80-b1ae-42ed-a89c-424b764b284f-kube-api-access-2l24n\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:56:39.700083 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.700045    2576 generic.go:356] "Generic (PLEG): container finished" podID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerID="2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d" exitCode=0
Sep 09 20:56:39.700259 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.700088    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"
Sep 09 20:56:39.700259 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.700114    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" event={"ID":"745d7c80-b1ae-42ed-a89c-424b764b284f","Type":"ContainerDied","Data":"2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d"}
Sep 09 20:56:39.700259 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.700179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh" event={"ID":"745d7c80-b1ae-42ed-a89c-424b764b284f","Type":"ContainerDied","Data":"1287fbfcfded7142cd662bfa14df4ef7afa5a32ed148eaff6796e4238520f94c"}
Sep 09 20:56:39.700259 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.700204    2576 scope.go:117] "RemoveContainer" containerID="2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d"
Sep 09 20:56:39.709435 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.709415    2576 scope.go:117] "RemoveContainer" containerID="2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d"
Sep 09 20:56:39.709701 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:56:39.709679    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d\": container with ID starting with 2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d not found: ID does not exist" containerID="2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d"
Sep 09 20:56:39.709759 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.709708    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d"} err="failed to get container status \"2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d\": rpc error: code = NotFound desc = could not find container \"2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d\": container with ID starting with 2cc699cd48c523000fc55e6d72e62b408c54e56c98eed98b6e4b25ac8b931a7d not found: ID does not exist"
Sep 09 20:56:39.722534 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.722505    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"]
Sep 09 20:56:39.726579 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.726556    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-965sh"]
Sep 09 20:56:39.961773 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:39.961694    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" path="/var/lib/kubelet/pods/745d7c80-b1ae-42ed-a89c-424b764b284f/volumes"
Sep 09 20:56:44.999640 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:44.999604    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:47.440625 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:47.440584    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main" containerID="cri-o://2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7" gracePeriod=45
Sep 09 20:56:54.999656 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:54.999618    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:56:55.000122 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:56:54.999730    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:57:04.999825 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:04.999781    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:57:15.000004 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:14.999936    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 20:57:17.694068 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.694044    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:57:17.705488 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.705443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fdf39ca-9420-4c22-b111-3567ad1d5869-tls-certs\") pod \"9fdf39ca-9420-4c22-b111-3567ad1d5869\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") "
Sep 09 20:57:17.705579 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.705504    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n49s\" (UniqueName: \"kubernetes.io/projected/9fdf39ca-9420-4c22-b111-3567ad1d5869-kube-api-access-5n49s\") pod \"9fdf39ca-9420-4c22-b111-3567ad1d5869\" (UID: \"9fdf39ca-9420-4c22-b111-3567ad1d5869\") "
Sep 09 20:57:17.707798 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.707771    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fdf39ca-9420-4c22-b111-3567ad1d5869-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9fdf39ca-9420-4c22-b111-3567ad1d5869" (UID: "9fdf39ca-9420-4c22-b111-3567ad1d5869"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:57:17.707894 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.707770    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fdf39ca-9420-4c22-b111-3567ad1d5869-kube-api-access-5n49s" (OuterVolumeSpecName: "kube-api-access-5n49s") pod "9fdf39ca-9420-4c22-b111-3567ad1d5869" (UID: "9fdf39ca-9420-4c22-b111-3567ad1d5869"). InnerVolumeSpecName "kube-api-access-5n49s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:57:17.806386 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.806350    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9fdf39ca-9420-4c22-b111-3567ad1d5869-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:57:17.806386 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.806385    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n49s\" (UniqueName: \"kubernetes.io/projected/9fdf39ca-9420-4c22-b111-3567ad1d5869-kube-api-access-5n49s\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 20:57:17.878010 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.877975    2576 generic.go:356] "Generic (PLEG): container finished" podID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerID="2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7" exitCode=0
Sep 09 20:57:17.878174 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.878017    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"
Sep 09 20:57:17.878174 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.878026    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" event={"ID":"9fdf39ca-9420-4c22-b111-3567ad1d5869","Type":"ContainerDied","Data":"2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7"}
Sep 09 20:57:17.878174 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.878077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x" event={"ID":"9fdf39ca-9420-4c22-b111-3567ad1d5869","Type":"ContainerDied","Data":"8074ca6a331779fd7890d2473cafa37c66fca398286f597072309f5429532b7c"}
Sep 09 20:57:17.878174 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.878098    2576 scope.go:117] "RemoveContainer" containerID="2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7"
Sep 09 20:57:17.887301 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.887279    2576 scope.go:117] "RemoveContainer" containerID="2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7"
Sep 09 20:57:17.887563 ip-10-0-135-242 kubenswrapper[2576]: E0909 20:57:17.887542    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7\": container with ID starting with 2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7 not found: ID does not exist" containerID="2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7"
Sep 09 20:57:17.887630 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.887576    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7"} err="failed to get container status \"2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7\": rpc error: code = NotFound desc = could not find container \"2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7\": container with ID starting with 2a7f881d6667defd8f822252429f4732ed0462184b1efde433ab002aac4e8cd7 not found: ID does not exist"
Sep 09 20:57:17.901588 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.901561    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"]
Sep 09 20:57:17.913441 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.913419    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-scheddc6x"]
Sep 09 20:57:17.962455 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:17.962381    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" path="/var/lib/kubelet/pods/9fdf39ca-9420-4c22-b111-3567ad1d5869/volumes"
Sep 09 20:57:43.540264 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540224    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"]
Sep 09 20:57:43.540797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540552    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main"
Sep 09 20:57:43.540797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540563    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main"
Sep 09 20:57:43.540797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540573    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main"
Sep 09 20:57:43.540797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540578    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main"
Sep 09 20:57:43.540797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540634    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fdf39ca-9420-4c22-b111-3567ad1d5869" containerName="main"
Sep 09 20:57:43.540797 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.540644    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="745d7c80-b1ae-42ed-a89c-424b764b284f" containerName="main"
Sep 09 20:57:43.543379 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.543362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.546834 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.546795    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 09 20:57:43.546994 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.546889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cwr8d\""
Sep 09 20:57:43.546994 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.546889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 09 20:57:43.546994 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.546890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 09 20:57:43.547184 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.547000    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-4k9k7\""
Sep 09 20:57:43.552695 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.552670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"]
Sep 09 20:57:43.614429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.614386    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c51d27-9305-4673-8e36-838418db762e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.614429 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.614426    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwp8s\" (UniqueName: \"kubernetes.io/projected/94c51d27-9305-4673-8e36-838418db762e-kube-api-access-rwp8s\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.715632 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.715595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c51d27-9305-4673-8e36-838418db762e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.715632 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.715632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwp8s\" (UniqueName: \"kubernetes.io/projected/94c51d27-9305-4673-8e36-838418db762e-kube-api-access-rwp8s\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.718339 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.718310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c51d27-9305-4673-8e36-838418db762e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.724070 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.724046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwp8s\" (UniqueName: \"kubernetes.io/projected/94c51d27-9305-4673-8e36-838418db762e-kube-api-access-rwp8s\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:43.854327 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:43.854293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:44.129630 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:44.129554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"]
Sep 09 20:57:44.131434 ip-10-0-135-242 kubenswrapper[2576]: W0909 20:57:44.131399    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c51d27_9305_4673_8e36_838418db762e.slice/crio-77e1ba23617911fb48d653cc0c39c20fe6e31e44c45a1cf6b07901a6e883f58e WatchSource:0}: Error finding container 77e1ba23617911fb48d653cc0c39c20fe6e31e44c45a1cf6b07901a6e883f58e: Status 404 returned error can't find the container with id 77e1ba23617911fb48d653cc0c39c20fe6e31e44c45a1cf6b07901a6e883f58e
Sep 09 20:57:45.004134 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:45.004100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" event={"ID":"94c51d27-9305-4673-8e36-838418db762e","Type":"ContainerStarted","Data":"35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686"}
Sep 09 20:57:45.004552 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:45.004149    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" event={"ID":"94c51d27-9305-4673-8e36-838418db762e","Type":"ContainerStarted","Data":"77e1ba23617911fb48d653cc0c39c20fe6e31e44c45a1cf6b07901a6e883f58e"}
Sep 09 20:57:45.004552 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:45.004234    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:57:45.022538 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:57:45.022489    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podStartSLOduration=2.022474084 podStartE2EDuration="2.022474084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:45.020729856 +0000 UTC m=+3767.538867384" watchObservedRunningTime="2026-09-09 20:57:45.022474084 +0000 UTC m=+3767.540611610"
Sep 09 20:58:16.011124 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:58:16.011094    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 20:59:58.269240 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:59:58.269208    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 20:59:58.281596 ip-10-0-135-242 kubenswrapper[2576]: I0909 20:59:58.281572    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 21:00:03.216867 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:03.216832    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"]
Sep 09 21:00:06.010808 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:06.010764    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:16.010767 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:16.010724    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:18.236661 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:18.236590    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main" containerID="cri-o://35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686" gracePeriod=45
Sep 09 21:00:26.011055 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:26.011017    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:26.011461 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:26.011127    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 21:00:36.010828 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:36.010781    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:46.010858 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:46.010815    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:00:48.487066 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.487034    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 21:00:48.525378 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.525295    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c51d27-9305-4673-8e36-838418db762e-tls-certs\") pod \"94c51d27-9305-4673-8e36-838418db762e\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") "
Sep 09 21:00:48.525544 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.525402    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwp8s\" (UniqueName: \"kubernetes.io/projected/94c51d27-9305-4673-8e36-838418db762e-kube-api-access-rwp8s\") pod \"94c51d27-9305-4673-8e36-838418db762e\" (UID: \"94c51d27-9305-4673-8e36-838418db762e\") "
Sep 09 21:00:48.527694 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.527665    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c51d27-9305-4673-8e36-838418db762e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "94c51d27-9305-4673-8e36-838418db762e" (UID: "94c51d27-9305-4673-8e36-838418db762e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:48.527694 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.527681    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c51d27-9305-4673-8e36-838418db762e-kube-api-access-rwp8s" (OuterVolumeSpecName: "kube-api-access-rwp8s") pod "94c51d27-9305-4673-8e36-838418db762e" (UID: "94c51d27-9305-4673-8e36-838418db762e"). InnerVolumeSpecName "kube-api-access-rwp8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:48.625945 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.625897    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c51d27-9305-4673-8e36-838418db762e-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:00:48.625945 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.625938    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwp8s\" (UniqueName: \"kubernetes.io/projected/94c51d27-9305-4673-8e36-838418db762e-kube-api-access-rwp8s\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:00:48.885094 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.885063    2576 generic.go:356] "Generic (PLEG): container finished" podID="94c51d27-9305-4673-8e36-838418db762e" containerID="35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686" exitCode=0
Sep 09 21:00:48.885273 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.885109    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"
Sep 09 21:00:48.885273 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.885117    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" event={"ID":"94c51d27-9305-4673-8e36-838418db762e","Type":"ContainerDied","Data":"35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686"}
Sep 09 21:00:48.885273 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.885186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z" event={"ID":"94c51d27-9305-4673-8e36-838418db762e","Type":"ContainerDied","Data":"77e1ba23617911fb48d653cc0c39c20fe6e31e44c45a1cf6b07901a6e883f58e"}
Sep 09 21:00:48.885273 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.885209    2576 scope.go:117] "RemoveContainer" containerID="35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686"
Sep 09 21:00:48.894319 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.894299    2576 scope.go:117] "RemoveContainer" containerID="35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686"
Sep 09 21:00:48.894563 ip-10-0-135-242 kubenswrapper[2576]: E0909 21:00:48.894546    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686\": container with ID starting with 35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686 not found: ID does not exist" containerID="35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686"
Sep 09 21:00:48.894612 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.894571    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686"} err="failed to get container status \"35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686\": rpc error: code = NotFound desc = could not find container \"35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686\": container with ID starting with 35e9baca33d56ec6d159bca54a4b47c70ae2a2e0fcf598d9beeedffdbad45686 not found: ID does not exist"
Sep 09 21:00:48.915395 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.915370    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"]
Sep 09 21:00:48.923491 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:48.923469    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c48hw75z"]
Sep 09 21:00:49.961194 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:00:49.961163    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94c51d27-9305-4673-8e36-838418db762e" path="/var/lib/kubelet/pods/94c51d27-9305-4673-8e36-838418db762e/volumes"
Sep 09 21:01:07.316657 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.316621    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"]
Sep 09 21:01:07.317107 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.316940    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main"
Sep 09 21:01:07.317107 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.316965    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main"
Sep 09 21:01:07.317107 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.317014    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="94c51d27-9305-4673-8e36-838418db762e" containerName="main"
Sep 09 21:01:07.321130 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.321112    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.325124 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.325098    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 09 21:01:07.325249 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.325140    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 09 21:01:07.325249 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.325166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-thxvh\""
Sep 09 21:01:07.325249 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.325141    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-tzpbg\""
Sep 09 21:01:07.325249 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.325224    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 09 21:01:07.334265 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.334240    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"]
Sep 09 21:01:07.371523 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.371491    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.371710 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.371548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45lg4\" (UniqueName: \"kubernetes.io/projected/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-kube-api-access-45lg4\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.472741 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.472700    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45lg4\" (UniqueName: \"kubernetes.io/projected/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-kube-api-access-45lg4\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.472979 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.472777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.475499 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.475470    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.485465 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.485434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45lg4\" (UniqueName: \"kubernetes.io/projected/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-kube-api-access-45lg4\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.631007 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.630947    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.905485 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.905454    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"]
Sep 09 21:01:07.907350 ip-10-0-135-242 kubenswrapper[2576]: W0909 21:01:07.907318    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b1b63b_ff0f_4b68_bae5_4f73166b18b4.slice/crio-5ab440b693767cde8fac0ac465d7878fd63456deab427a70d516a29da27dc89e WatchSource:0}: Error finding container 5ab440b693767cde8fac0ac465d7878fd63456deab427a70d516a29da27dc89e: Status 404 returned error can't find the container with id 5ab440b693767cde8fac0ac465d7878fd63456deab427a70d516a29da27dc89e
Sep 09 21:01:07.977582 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.977542    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" event={"ID":"84b1b63b-ff0f-4b68-bae5-4f73166b18b4","Type":"ContainerStarted","Data":"79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3"}
Sep 09 21:01:07.977756 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.977615    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:01:07.977756 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.977640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" event={"ID":"84b1b63b-ff0f-4b68-bae5-4f73166b18b4","Type":"ContainerStarted","Data":"5ab440b693767cde8fac0ac465d7878fd63456deab427a70d516a29da27dc89e"}
Sep 09 21:01:07.995651 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:07.995598    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podStartSLOduration=0.995581441 podStartE2EDuration="995.581441ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:07.993886422 +0000 UTC m=+3970.512023964" watchObservedRunningTime="2026-09-09 21:01:07.995581441 +0000 UTC m=+3970.513718968"
Sep 09 21:01:38.983513 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:01:38.983480    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:02:07.215087 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.215053    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"]
Sep 09 21:02:07.218929 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.218905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.223736 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.223709    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-p5zfq\""
Sep 09 21:02:07.223736 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.223727    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 09 21:02:07.223927 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.223726    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 09 21:02:07.224177 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.224153    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-rvvb9\""
Sep 09 21:02:07.224298 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.224214    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 09 21:02:07.230347 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.230325    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"]
Sep 09 21:02:07.257128 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.257101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74fdc60a-12c4-4085-b117-2bbcea5ca663-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.257285 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.257150    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cts4x\" (UniqueName: \"kubernetes.io/projected/74fdc60a-12c4-4085-b117-2bbcea5ca663-kube-api-access-cts4x\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.357579 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.357541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74fdc60a-12c4-4085-b117-2bbcea5ca663-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.357766 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.357597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cts4x\" (UniqueName: \"kubernetes.io/projected/74fdc60a-12c4-4085-b117-2bbcea5ca663-kube-api-access-cts4x\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.360422 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.360399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74fdc60a-12c4-4085-b117-2bbcea5ca663-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.366535 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.366505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cts4x\" (UniqueName: \"kubernetes.io/projected/74fdc60a-12c4-4085-b117-2bbcea5ca663-kube-api-access-cts4x\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.531931 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.531822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:07.808510 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:07.808475    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"]
Sep 09 21:02:07.810367 ip-10-0-135-242 kubenswrapper[2576]: W0909 21:02:07.810328    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fdc60a_12c4_4085_b117_2bbcea5ca663.slice/crio-eed418d95a957ca71b8573696fd9d5de03ac35a49048713c64c6901ba4b1c01b WatchSource:0}: Error finding container eed418d95a957ca71b8573696fd9d5de03ac35a49048713c64c6901ba4b1c01b: Status 404 returned error can't find the container with id eed418d95a957ca71b8573696fd9d5de03ac35a49048713c64c6901ba4b1c01b
Sep 09 21:02:08.267800 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:08.267760    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" event={"ID":"74fdc60a-12c4-4085-b117-2bbcea5ca663","Type":"ContainerStarted","Data":"43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2"}
Sep 09 21:02:08.268344 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:08.267812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" event={"ID":"74fdc60a-12c4-4085-b117-2bbcea5ca663","Type":"ContainerStarted","Data":"eed418d95a957ca71b8573696fd9d5de03ac35a49048713c64c6901ba4b1c01b"}
Sep 09 21:02:08.268344 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:08.267885    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:02:08.302434 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:08.302372    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" podStartSLOduration=1.30235214 podStartE2EDuration="1.30235214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:08.301791352 +0000 UTC m=+4030.819928879" watchObservedRunningTime="2026-09-09 21:02:08.30235214 +0000 UTC m=+4030.820489670"
Sep 09 21:02:39.275408 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:02:39.275335    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:03:56.446086 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:03:56.445992    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"]
Sep 09 21:03:58.983550 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:03:58.983506    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:08.984253 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:08.984217    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:11.466781 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:11.466733    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main" containerID="cri-o://79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3" gracePeriod=45
Sep 09 21:04:18.984386 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:18.984348    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:18.984776 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:18.984450    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:04:22.615706 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:22.615671    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"]
Sep 09 21:04:28.983901 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:28.983861    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:29.275722 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:29.275617    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:37.636530 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:37.636487    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main" containerID="cri-o://43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2" gracePeriod=45
Sep 09 21:04:38.983342 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:38.983299    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:39.275330 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:39.275243    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:41.723419 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.723394    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:04:41.849405 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.849369    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45lg4\" (UniqueName: \"kubernetes.io/projected/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-kube-api-access-45lg4\") pod \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") "
Sep 09 21:04:41.849565 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.849433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-tls-certs\") pod \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\" (UID: \"84b1b63b-ff0f-4b68-bae5-4f73166b18b4\") "
Sep 09 21:04:41.851747 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.851710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84b1b63b-ff0f-4b68-bae5-4f73166b18b4" (UID: "84b1b63b-ff0f-4b68-bae5-4f73166b18b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:41.851860 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.851718    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-kube-api-access-45lg4" (OuterVolumeSpecName: "kube-api-access-45lg4") pod "84b1b63b-ff0f-4b68-bae5-4f73166b18b4" (UID: "84b1b63b-ff0f-4b68-bae5-4f73166b18b4"). InnerVolumeSpecName "kube-api-access-45lg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:41.949982 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.949931    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45lg4\" (UniqueName: \"kubernetes.io/projected/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-kube-api-access-45lg4\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:04:41.950153 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.949999    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b1b63b-ff0f-4b68-bae5-4f73166b18b4-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:04:41.992855 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.992823    2576 generic.go:356] "Generic (PLEG): container finished" podID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerID="79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3" exitCode=0
Sep 09 21:04:41.993028 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.992878    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"
Sep 09 21:04:41.993028 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.992911    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" event={"ID":"84b1b63b-ff0f-4b68-bae5-4f73166b18b4","Type":"ContainerDied","Data":"79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3"}
Sep 09 21:04:41.993028 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.992986    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm" event={"ID":"84b1b63b-ff0f-4b68-bae5-4f73166b18b4","Type":"ContainerDied","Data":"5ab440b693767cde8fac0ac465d7878fd63456deab427a70d516a29da27dc89e"}
Sep 09 21:04:41.993028 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:41.993011    2576 scope.go:117] "RemoveContainer" containerID="79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3"
Sep 09 21:04:42.001808 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:42.001792    2576 scope.go:117] "RemoveContainer" containerID="79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3"
Sep 09 21:04:42.002114 ip-10-0-135-242 kubenswrapper[2576]: E0909 21:04:42.002092    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3\": container with ID starting with 79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3 not found: ID does not exist" containerID="79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3"
Sep 09 21:04:42.002170 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:42.002119    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3"} err="failed to get container status \"79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3\": rpc error: code = NotFound desc = could not find container \"79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3\": container with ID starting with 79e886e71c4647f28e586423fb0fa9e7f1304080b660381d68fc9755bf5275a3 not found: ID does not exist"
Sep 09 21:04:42.010005 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:42.009980    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"]
Sep 09 21:04:42.014103 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:42.014080    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b4594dgm"]
Sep 09 21:04:43.962234 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:43.962201    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" path="/var/lib/kubelet/pods/84b1b63b-ff0f-4b68-bae5-4f73166b18b4/volumes"
Sep 09 21:04:49.275814 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:49.275774    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:04:49.276217 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:49.275878    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:04:52.355818 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.355786    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"]
Sep 09 21:04:52.356321 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.356139    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main"
Sep 09 21:04:52.356321 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.356152    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main"
Sep 09 21:04:52.356321 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.356217    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="84b1b63b-ff0f-4b68-bae5-4f73166b18b4" containerName="main"
Sep 09 21:04:52.358835 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.358815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.362555 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.362522    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 09 21:04:52.362555 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.362543    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 09 21:04:52.362785 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.362559    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 09 21:04:52.362785 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.362663    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-6wvcw\""
Sep 09 21:04:52.362785 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.362676    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-4wf97\""
Sep 09 21:04:52.371600 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.371573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"]
Sep 09 21:04:52.443672 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.443634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkllc\" (UniqueName: \"kubernetes.io/projected/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-kube-api-access-gkllc\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.443866 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.443689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.545038 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.545001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.545221 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.545072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkllc\" (UniqueName: \"kubernetes.io/projected/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-kube-api-access-gkllc\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.547811 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.547784    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.554346 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.554325    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkllc\" (UniqueName: \"kubernetes.io/projected/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-kube-api-access-gkllc\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.671595 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.671482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:52.952574 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:52.952546    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"]
Sep 09 21:04:52.955053 ip-10-0-135-242 kubenswrapper[2576]: W0909 21:04:52.955023    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b1fe6cc_4b29_48aa_a48b_10d1caaac48a.slice/crio-7350db4f1da5aaf5fbfb7e27289cca01dbd7c7c9c2920f9c22ba12fd90729d7e WatchSource:0}: Error finding container 7350db4f1da5aaf5fbfb7e27289cca01dbd7c7c9c2920f9c22ba12fd90729d7e: Status 404 returned error can't find the container with id 7350db4f1da5aaf5fbfb7e27289cca01dbd7c7c9c2920f9c22ba12fd90729d7e
Sep 09 21:04:53.049782 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:53.049747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" event={"ID":"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a","Type":"ContainerStarted","Data":"f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da"}
Sep 09 21:04:53.049996 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:53.049798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" event={"ID":"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a","Type":"ContainerStarted","Data":"7350db4f1da5aaf5fbfb7e27289cca01dbd7c7c9c2920f9c22ba12fd90729d7e"}
Sep 09 21:04:53.049996 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:53.049819    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:04:58.303050 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:58.303013    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 21:04:58.316112 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:58.316088    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 21:04:59.275853 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:04:59.275813    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:05:07.874606 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:07.874581    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:05:07.896072 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:07.896007    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podStartSLOduration=15.895987128 podStartE2EDuration="15.895987128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:53.066993399 +0000 UTC m=+4195.585130933" watchObservedRunningTime="2026-09-09 21:05:07.895987128 +0000 UTC m=+4210.414124655"
Sep 09 21:05:08.070180 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.070089    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cts4x\" (UniqueName: \"kubernetes.io/projected/74fdc60a-12c4-4085-b117-2bbcea5ca663-kube-api-access-cts4x\") pod \"74fdc60a-12c4-4085-b117-2bbcea5ca663\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") "
Sep 09 21:05:08.070180 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.070162    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74fdc60a-12c4-4085-b117-2bbcea5ca663-tls-certs\") pod \"74fdc60a-12c4-4085-b117-2bbcea5ca663\" (UID: \"74fdc60a-12c4-4085-b117-2bbcea5ca663\") "
Sep 09 21:05:08.072596 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.072562    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fdc60a-12c4-4085-b117-2bbcea5ca663-kube-api-access-cts4x" (OuterVolumeSpecName: "kube-api-access-cts4x") pod "74fdc60a-12c4-4085-b117-2bbcea5ca663" (UID: "74fdc60a-12c4-4085-b117-2bbcea5ca663"). InnerVolumeSpecName "kube-api-access-cts4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:08.072596 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.072581    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fdc60a-12c4-4085-b117-2bbcea5ca663-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "74fdc60a-12c4-4085-b117-2bbcea5ca663" (UID: "74fdc60a-12c4-4085-b117-2bbcea5ca663"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:05:08.127303 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.127264    2576 generic.go:356] "Generic (PLEG): container finished" podID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerID="43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2" exitCode=0
Sep 09 21:05:08.127469 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.127312    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"
Sep 09 21:05:08.127469 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.127353    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" event={"ID":"74fdc60a-12c4-4085-b117-2bbcea5ca663","Type":"ContainerDied","Data":"43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2"}
Sep 09 21:05:08.127469 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.127410    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt" event={"ID":"74fdc60a-12c4-4085-b117-2bbcea5ca663","Type":"ContainerDied","Data":"eed418d95a957ca71b8573696fd9d5de03ac35a49048713c64c6901ba4b1c01b"}
Sep 09 21:05:08.127469 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.127437    2576 scope.go:117] "RemoveContainer" containerID="43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2"
Sep 09 21:05:08.136178 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.136159    2576 scope.go:117] "RemoveContainer" containerID="43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2"
Sep 09 21:05:08.136459 ip-10-0-135-242 kubenswrapper[2576]: E0909 21:05:08.136441    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2\": container with ID starting with 43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2 not found: ID does not exist" containerID="43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2"
Sep 09 21:05:08.136526 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.136466    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2"} err="failed to get container status \"43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2\": rpc error: code = NotFound desc = could not find container \"43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2\": container with ID starting with 43809e180434a2dd604ceb7d76cfe183e38ddd5ac90a2f62d12c5cef9aae65c2 not found: ID does not exist"
Sep 09 21:05:08.150132 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.150101    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"]
Sep 09 21:05:08.153411 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.153389    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-q74nt"]
Sep 09 21:05:08.170793 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.170767    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74fdc60a-12c4-4085-b117-2bbcea5ca663-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:05:08.170793 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:08.170791    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cts4x\" (UniqueName: \"kubernetes.io/projected/74fdc60a-12c4-4085-b117-2bbcea5ca663-kube-api-access-cts4x\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:05:09.963631 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:09.963583    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" path="/var/lib/kubelet/pods/74fdc60a-12c4-4085-b117-2bbcea5ca663/volumes"
Sep 09 21:05:24.056988 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:24.056898    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:05:31.971867 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.971837    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"]
Sep 09 21:05:31.972401 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.972176    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main"
Sep 09 21:05:31.972401 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.972189    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main"
Sep 09 21:05:31.972401 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.972238    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fdc60a-12c4-4085-b117-2bbcea5ca663" containerName="main"
Sep 09 21:05:31.974212 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.974197    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:31.977906 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.977872    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 09 21:05:31.978248 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.978227    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-dfg67\""
Sep 09 21:05:31.978353 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.978257    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-hf426\""
Sep 09 21:05:31.978738 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.978721    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 09 21:05:31.978920 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.978902    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 09 21:05:31.990894 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:31.990867    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"]
Sep 09 21:05:32.042418 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.042382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gwwg\" (UniqueName: \"kubernetes.io/projected/230312a9-3313-486c-8438-2ad9fe3c0583-kube-api-access-7gwwg\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.042589 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.042452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230312a9-3313-486c-8438-2ad9fe3c0583-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.142879 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.142845    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230312a9-3313-486c-8438-2ad9fe3c0583-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.143092 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.142893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gwwg\" (UniqueName: \"kubernetes.io/projected/230312a9-3313-486c-8438-2ad9fe3c0583-kube-api-access-7gwwg\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.145475 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.145444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230312a9-3313-486c-8438-2ad9fe3c0583-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.151922 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.151903    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gwwg\" (UniqueName: \"kubernetes.io/projected/230312a9-3313-486c-8438-2ad9fe3c0583-kube-api-access-7gwwg\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.284288 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.284197    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:32.557623 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:32.557597    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"]
Sep 09 21:05:32.559927 ip-10-0-135-242 kubenswrapper[2576]: W0909 21:05:32.559899    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230312a9_3313_486c_8438_2ad9fe3c0583.slice/crio-99bcb5ff775e84b4961da8d4e837bf5ef3319ce425d37097df18f06907935944 WatchSource:0}: Error finding container 99bcb5ff775e84b4961da8d4e837bf5ef3319ce425d37097df18f06907935944: Status 404 returned error can't find the container with id 99bcb5ff775e84b4961da8d4e837bf5ef3319ce425d37097df18f06907935944
Sep 09 21:05:33.246609 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:33.246575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" event={"ID":"230312a9-3313-486c-8438-2ad9fe3c0583","Type":"ContainerStarted","Data":"8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a"}
Sep 09 21:05:33.247058 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:33.246627    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" event={"ID":"230312a9-3313-486c-8438-2ad9fe3c0583","Type":"ContainerStarted","Data":"99bcb5ff775e84b4961da8d4e837bf5ef3319ce425d37097df18f06907935944"}
Sep 09 21:05:33.247058 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:33.246706    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:05:33.265977 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:05:33.265902    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podStartSLOduration=2.265886693 podStartE2EDuration="2.265886693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:33.263801782 +0000 UTC m=+4235.781939322" watchObservedRunningTime="2026-09-09 21:05:33.265886693 +0000 UTC m=+4235.784024219"
Sep 09 21:06:04.253570 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:04.253531    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:06:10.894997 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:10.894930    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"]
Sep 09 21:06:14.254189 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:14.254154    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:06:24.254116 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:24.254076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:06:25.915701 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:25.915635    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main" containerID="cri-o://8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a" gracePeriod=45
Sep 09 21:06:34.254010 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:34.253929    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:06:34.254450 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:34.254074    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:06:44.254163 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:44.254119    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:06:54.253425 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:54.253330    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:06:56.178588 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.178520    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:06:56.240462 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.240421    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gwwg\" (UniqueName: \"kubernetes.io/projected/230312a9-3313-486c-8438-2ad9fe3c0583-kube-api-access-7gwwg\") pod \"230312a9-3313-486c-8438-2ad9fe3c0583\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") "
Sep 09 21:06:56.240678 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.240522    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230312a9-3313-486c-8438-2ad9fe3c0583-tls-certs\") pod \"230312a9-3313-486c-8438-2ad9fe3c0583\" (UID: \"230312a9-3313-486c-8438-2ad9fe3c0583\") "
Sep 09 21:06:56.242967 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.242917    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/230312a9-3313-486c-8438-2ad9fe3c0583-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "230312a9-3313-486c-8438-2ad9fe3c0583" (UID: "230312a9-3313-486c-8438-2ad9fe3c0583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:56.243092 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.242927    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230312a9-3313-486c-8438-2ad9fe3c0583-kube-api-access-7gwwg" (OuterVolumeSpecName: "kube-api-access-7gwwg") pod "230312a9-3313-486c-8438-2ad9fe3c0583" (UID: "230312a9-3313-486c-8438-2ad9fe3c0583"). InnerVolumeSpecName "kube-api-access-7gwwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:56.341641 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.341601    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gwwg\" (UniqueName: \"kubernetes.io/projected/230312a9-3313-486c-8438-2ad9fe3c0583-kube-api-access-7gwwg\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:06:56.341641 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.341635    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/230312a9-3313-486c-8438-2ad9fe3c0583-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:06:56.654506 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.654469    2576 generic.go:356] "Generic (PLEG): container finished" podID="230312a9-3313-486c-8438-2ad9fe3c0583" containerID="8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a" exitCode=0
Sep 09 21:06:56.654691 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.654516    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"
Sep 09 21:06:56.654691 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.654558    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" event={"ID":"230312a9-3313-486c-8438-2ad9fe3c0583","Type":"ContainerDied","Data":"8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a"}
Sep 09 21:06:56.654691 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.654607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc" event={"ID":"230312a9-3313-486c-8438-2ad9fe3c0583","Type":"ContainerDied","Data":"99bcb5ff775e84b4961da8d4e837bf5ef3319ce425d37097df18f06907935944"}
Sep 09 21:06:56.654691 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.654634    2576 scope.go:117] "RemoveContainer" containerID="8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a"
Sep 09 21:06:56.663754 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.663736    2576 scope.go:117] "RemoveContainer" containerID="8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a"
Sep 09 21:06:56.664075 ip-10-0-135-242 kubenswrapper[2576]: E0909 21:06:56.664056    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a\": container with ID starting with 8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a not found: ID does not exist" containerID="8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a"
Sep 09 21:06:56.664134 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.664084    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a"} err="failed to get container status \"8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a\": rpc error: code = NotFound desc = could not find container \"8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a\": container with ID starting with 8d4013a67b447691d728c94bf13a914f507e9129169d3a65738ee55b7e664a2a not found: ID does not exist"
Sep 09 21:06:56.677730 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.677698    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"]
Sep 09 21:06:56.683119 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:56.683094    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-98xjc"]
Sep 09 21:06:57.962512 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:06:57.962475    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" path="/var/lib/kubelet/pods/230312a9-3313-486c-8438-2ad9fe3c0583/volumes"
Sep 09 21:07:44.057419 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:07:44.057382    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:44.087566 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:07:44.087519    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"]
Sep 09 21:07:54.056186 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:07:54.056144    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:07:59.109896 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:07:59.109849    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main" containerID="cri-o://f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da" gracePeriod=45
Sep 09 21:08:04.056839 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:04.056795    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:04.057331 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:04.056901    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:08:14.056636 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:14.056600    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:24.057150 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:24.057060    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 09 21:08:29.365537 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.365503    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:08:29.502175 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.502080    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-tls-certs\") pod \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") "
Sep 09 21:08:29.502175 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.502146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkllc\" (UniqueName: \"kubernetes.io/projected/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-kube-api-access-gkllc\") pod \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\" (UID: \"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a\") "
Sep 09 21:08:29.504517 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.504488    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" (UID: "1b1fe6cc-4b29-48aa-a48b-10d1caaac48a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:08:29.504631 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.504562    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-kube-api-access-gkllc" (OuterVolumeSpecName: "kube-api-access-gkllc") pod "1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" (UID: "1b1fe6cc-4b29-48aa-a48b-10d1caaac48a"). InnerVolumeSpecName "kube-api-access-gkllc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:29.603145 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.603107    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-tls-certs\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:08:29.603145 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:29.603141    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkllc\" (UniqueName: \"kubernetes.io/projected/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a-kube-api-access-gkllc\") on node \"ip-10-0-135-242.ec2.internal\" DevicePath \"\""
Sep 09 21:08:30.108302 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.108265    2576 generic.go:356] "Generic (PLEG): container finished" podID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerID="f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da" exitCode=0
Sep 09 21:08:30.108496 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.108311    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"
Sep 09 21:08:30.108496 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.108354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" event={"ID":"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a","Type":"ContainerDied","Data":"f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da"}
Sep 09 21:08:30.108496 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.108407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df" event={"ID":"1b1fe6cc-4b29-48aa-a48b-10d1caaac48a","Type":"ContainerDied","Data":"7350db4f1da5aaf5fbfb7e27289cca01dbd7c7c9c2920f9c22ba12fd90729d7e"}
Sep 09 21:08:30.108496 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.108428    2576 scope.go:117] "RemoveContainer" containerID="f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da"
Sep 09 21:08:30.117383 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.117364    2576 scope.go:117] "RemoveContainer" containerID="f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da"
Sep 09 21:08:30.117680 ip-10-0-135-242 kubenswrapper[2576]: E0909 21:08:30.117659    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da\": container with ID starting with f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da not found: ID does not exist" containerID="f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da"
Sep 09 21:08:30.117757 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.117688    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da"} err="failed to get container status \"f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da\": rpc error: code = NotFound desc = could not find container \"f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da\": container with ID starting with f9285205c8944c0a531b94bfbf8c5def6412ae9282f7be3bb81c86f2c4d150da not found: ID does not exist"
Sep 09 21:08:30.132631 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.132590    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"]
Sep 09 21:08:30.135432 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:30.135409    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-ln9df"]
Sep 09 21:08:31.962335 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:31.962299    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" path="/var/lib/kubelet/pods/1b1fe6cc-4b29-48aa-a48b-10d1caaac48a/volumes"
Sep 09 21:08:56.042997 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:56.042945    2576 ???:1] "http: TLS handshake error from 10.0.141.196:38268: EOF"
Sep 09 21:08:56.143080 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:56.143049    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pcrbh_5ccd73c4-40c5-401e-8d91-7171e03ae27a/discovery/0.log"
Sep 09 21:08:56.187410 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:56.187382    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59dc87fb98-gmrgs_240f7196-4f55-4ab8-9503-877e99f7cf0d/router/0.log"
Sep 09 21:08:56.898120 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:56.898082    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pcrbh_5ccd73c4-40c5-401e-8d91-7171e03ae27a/discovery/0.log"
Sep 09 21:08:56.948465 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:56.948427    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59dc87fb98-gmrgs_240f7196-4f55-4ab8-9503-877e99f7cf0d/router/0.log"
Sep 09 21:08:57.987479 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:57.987448    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-2p6c6_c3390f21-57b3-4b82-9cf7-1a8ba4754abd/manager/0.log"
Sep 09 21:08:58.203632 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:08:58.203598    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-4gdsf_36c80767-9be6-4fc8-90be-8ca5b1ad360f/manager/0.log"
Sep 09 21:09:07.426816 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:07.426779    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mc9xw_9d1db42d-aaa6-4d16-92f1-4be2b6aaa988/global-pull-secret-syncer/0.log"
Sep 09 21:09:07.579202 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:07.579170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t9j5m_1f736b1b-7aa0-4509-a18b-c25c0ff131b6/konnectivity-agent/0.log"
Sep 09 21:09:07.660477 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:07.660440    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-242.ec2.internal_c620202158ca74d37111884761e47e70/haproxy/0.log"
Sep 09 21:09:12.985234 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:12.985203    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6fc8f864f4-2p6c6_c3390f21-57b3-4b82-9cf7-1a8ba4754abd/manager/0.log"
Sep 09 21:09:13.213679 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:13.213638    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-4gdsf_36c80767-9be6-4fc8-90be-8ca5b1ad360f/manager/0.log"
Sep 09 21:09:14.928188 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:14.928144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-t7rqt_d274f779-e7ee-4653-853b-632e5ef8926c/cluster-monitoring-operator/0.log"
Sep 09 21:09:15.317635 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:15.317536    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgff6_5d4092ca-5f57-4511-a10b-b85798dc89d5/node-exporter/0.log"
Sep 09 21:09:15.361998 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:15.361941    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgff6_5d4092ca-5f57-4511-a10b-b85798dc89d5/kube-rbac-proxy/0.log"
Sep 09 21:09:15.394248 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:15.394217    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgff6_5d4092ca-5f57-4511-a10b-b85798dc89d5/init-textfile/0.log"
Sep 09 21:09:17.691802 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.691761    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"]
Sep 09 21:09:17.692345 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.692129    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main"
Sep 09 21:09:17.692345 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.692146    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main"
Sep 09 21:09:17.692345 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.692177    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main"
Sep 09 21:09:17.692345 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.692184    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main"
Sep 09 21:09:17.692345 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.692237    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b1fe6cc-4b29-48aa-a48b-10d1caaac48a" containerName="main"
Sep 09 21:09:17.692345 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.692244    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="230312a9-3313-486c-8438-2ad9fe3c0583" containerName="main"
Sep 09 21:09:17.694799 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.694781    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.697752 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.697733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-g4ljz\"/\"default-dockercfg-mjs97\""
Sep 09 21:09:17.697870 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.697856    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ljz\"/\"openshift-service-ca.crt\""
Sep 09 21:09:17.698717 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.698700    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-g4ljz\"/\"kube-root-ca.crt\""
Sep 09 21:09:17.712171 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.712145    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"]
Sep 09 21:09:17.795825 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.795785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-proc\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.795825 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.795828    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-lib-modules\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.796095 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.795853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-podres\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.796095 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.795988    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrvt\" (UniqueName: \"kubernetes.io/projected/cefd17d2-e478-4d02-97b7-b191ba5113e9-kube-api-access-xfrvt\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.796095 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.796030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-sys\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897293 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-proc\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897293 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-lib-modules\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897314    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-podres\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrvt\" (UniqueName: \"kubernetes.io/projected/cefd17d2-e478-4d02-97b7-b191ba5113e9-kube-api-access-xfrvt\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-sys\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897383    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-proc\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-lib-modules\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-sys\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.897543 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.897517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cefd17d2-e478-4d02-97b7-b191ba5113e9-podres\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.906468 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.906446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrvt\" (UniqueName: \"kubernetes.io/projected/cefd17d2-e478-4d02-97b7-b191ba5113e9-kube-api-access-xfrvt\") pod \"perf-node-gather-daemonset-b7mcq\" (UID: \"cefd17d2-e478-4d02-97b7-b191ba5113e9\") " pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:17.962142 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:17.962065    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-8zw8s_e742f61e-59e8-4a45-b3e5-705965dbf9e4/networking-console-plugin/0.log"
Sep 09 21:09:18.005109 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:18.005075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:18.284305 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:18.284281    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"]
Sep 09 21:09:18.286034 ip-10-0-135-242 kubenswrapper[2576]: W0909 21:09:18.286006    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcefd17d2_e478_4d02_97b7_b191ba5113e9.slice/crio-0af8b35970849b25468a1e32167e71f297f0c3b0a1c266eff65e43c6f948fada WatchSource:0}: Error finding container 0af8b35970849b25468a1e32167e71f297f0c3b0a1c266eff65e43c6f948fada: Status 404 returned error can't find the container with id 0af8b35970849b25468a1e32167e71f297f0c3b0a1c266eff65e43c6f948fada
Sep 09 21:09:18.334617 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:18.334581    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq" event={"ID":"cefd17d2-e478-4d02-97b7-b191ba5113e9","Type":"ContainerStarted","Data":"0af8b35970849b25468a1e32167e71f297f0c3b0a1c266eff65e43c6f948fada"}
Sep 09 21:09:18.562945 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:18.562864    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/1.log"
Sep 09 21:09:18.568043 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:18.568015    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-pmctv_3c2c780d-970d-4944-874f-722ac1a52827/console-operator/2.log"
Sep 09 21:09:19.084653 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:19.084623    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-t4sjh_7d28ce97-a63d-43dd-9eec-05d56df229e6/download-server/0.log"
Sep 09 21:09:19.340480 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:19.340450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq" event={"ID":"cefd17d2-e478-4d02-97b7-b191ba5113e9","Type":"ContainerStarted","Data":"fddaaa340d6fa46af42ac93f05bd54672d0179030b317f1aad3c5b733f22283f"}
Sep 09 21:09:19.340624 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:19.340519    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:19.364424 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:19.364366    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq" podStartSLOduration=2.3643523269999998 podStartE2EDuration="2.364352327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:09:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:19.364081333 +0000 UTC m=+4461.882218862" watchObservedRunningTime="2026-09-09 21:09:19.364352327 +0000 UTC m=+4461.882489853"
Sep 09 21:09:20.992671 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:20.992635    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v4q9w_d3172166-9c66-4d90-bd65-34942329005a/dns/0.log"
Sep 09 21:09:21.038827 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:21.038797    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v4q9w_d3172166-9c66-4d90-bd65-34942329005a/kube-rbac-proxy/0.log"
Sep 09 21:09:21.070669 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:21.070638    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c7sbj_7b05865d-a410-494c-803b-b6f4f63195fc/dns-node-resolver/0.log"
Sep 09 21:09:21.749447 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:21.749419    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-588fdcbc55-94fkx_24dd073f-2ed3-4923-9ced-2016b4113203/registry/0.log"
Sep 09 21:09:21.825629 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:21.825600    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6xnjs_5b9f9f07-7d47-4c98-a206-f1273c62974e/node-ca/0.log"
Sep 09 21:09:23.001380 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:23.001341    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pcrbh_5ccd73c4-40c5-401e-8d91-7171e03ae27a/discovery/0.log"
Sep 09 21:09:23.076081 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:23.076049    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59dc87fb98-gmrgs_240f7196-4f55-4ab8-9503-877e99f7cf0d/router/0.log"
Sep 09 21:09:23.835667 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:23.835632    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhd5g_968b1d70-5306-4f75-9153-9334ea8c468e/serve-healthcheck-canary/0.log"
Sep 09 21:09:24.457442 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:24.457411    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7mpmf_d11d726f-0be5-407f-91e9-3e189e8ea536/kube-rbac-proxy/0.log"
Sep 09 21:09:24.496322 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:24.496291    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7mpmf_d11d726f-0be5-407f-91e9-3e189e8ea536/exporter/0.log"
Sep 09 21:09:24.527309 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:24.527280    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7mpmf_d11d726f-0be5-407f-91e9-3e189e8ea536/extractor/0.log"
Sep 09 21:09:25.354664 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:25.354636    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-g4ljz/perf-node-gather-daemonset-b7mcq"
Sep 09 21:09:28.505083 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:28.505047    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pcrbh_5ccd73c4-40c5-401e-8d91-7171e03ae27a/discovery/0.log"
Sep 09 21:09:28.549600 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:28.549498    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59dc87fb98-gmrgs_240f7196-4f55-4ab8-9503-877e99f7cf0d/router/0.log"
Sep 09 21:09:28.917530 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:28.917496    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-7676c798d9-sq6lc_7a6535f4-825c-4d3b-b10c-fd61b6f9ab58/manager/0.log"
Sep 09 21:09:29.237869 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:29.237832    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-59449d9fbd-kl5hd_92a1c2ba-add6-4467-877a-f2cb39701927/manager/0.log"
Sep 09 21:09:29.692261 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:29.692228    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pcrbh_5ccd73c4-40c5-401e-8d91-7171e03ae27a/discovery/0.log"
Sep 09 21:09:29.897938 ip-10-0-135-242 kubenswrapper[2576]: I0909 21:09:29.897913    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59dc87fb98-gmrgs_240f7196-4f55-4ab8-9503-877e99f7cf0d/router/0.log"