Sep 16 18:41:28.152321 ip-10-0-143-190 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:41:28.152333 ip-10-0-143-190 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:41:28.152341 ip-10-0-143-190 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:41:28.152491 ip-10-0-143-190 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:41:38.164014 ip-10-0-143-190 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:41:38.164032 ip-10-0-143-190 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot b329a8e942014721b7b0cdffcbccce09 --
Sep 16 18:43:35.771238 ip-10-0-143-190 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:43:36.173366 ip-10-0-143-190 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:36.173366 ip-10-0-143-190 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:43:36.173366 ip-10-0-143-190 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:36.173366 ip-10-0-143-190 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:43:36.173366 ip-10-0-143-190 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:43:36.174398 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.174317    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:43:36.178397 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178381    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:36.178397 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178397    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178404    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178409    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178413    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178416    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178419    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178422    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178425    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178428    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178430    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178433    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178436    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178438    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178441    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178443    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178452    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178455    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178459    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178461    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:36.178460 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178464    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178467    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178470    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178473    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178476    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178478    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178480    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178483    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178486    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178488    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178491    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178494    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178496    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178499    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178501    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178504    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178506    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178509    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178511    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178514    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:36.178929 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178518    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178520    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178523    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178525    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178528    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178530    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178533    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178535    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178538    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178541    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178543    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178546    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178549    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178551    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178554    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178557    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178559    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178562    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178564    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178567    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:36.179525 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178569    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178572    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178574    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178577    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178579    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178583    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178587    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178591    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178594    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178597    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178599    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178602    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178605    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178608    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178611    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178613    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178616    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178619    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178622    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:36.180035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178624    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178628    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178631    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178639    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178642    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178645    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178648    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178652    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178654    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178657    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178660    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178663    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178665    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178668    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178670    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178673    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178676    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178678    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178681    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.178684    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:36.180509 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179437    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179446    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179449    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179452    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179456    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179458    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179461    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179464    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179467    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179471    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179473    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179476    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179480    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179482    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179485    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179488    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179494    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179497    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179500    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179503    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:36.181008 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179506    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179509    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179512    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179514    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179517    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179520    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179522    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179525    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179528    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179530    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179533    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179536    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179539    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179542    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179546    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179549    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179552    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179555    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179558    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:36.181507 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179560    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179563    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179566    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179569    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179573    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179576    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179579    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179582    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179585    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179588    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179591    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179594    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179597    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179600    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179603    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179608    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179611    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179614    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179616    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179619    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:36.182009 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179622    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179624    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179627    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179630    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179632    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179636    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179639    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179642    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179645    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179647    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179650    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179652    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179656    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179658    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179662    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179665    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179668    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179670    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179673    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179675    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:36.182499 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179679    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179682    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179684    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179687    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179690    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179693    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179695    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179698    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179701    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179703    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179705    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179708    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179710    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179713    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179716    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179718    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179721    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179724    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179726    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.179729    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:36.183018 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180707    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180714    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180718    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180721    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180725    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180728    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180731    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180734    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180737    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180740    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180743    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180746    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180748    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180751    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180754    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180757    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180759    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180762    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180765    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180768    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:36.183501 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180770    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180773    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180775    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180778    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180781    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180783    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180785    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180788    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180791    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180793    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180796    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180798    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180801    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180804    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180807    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180809    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180812    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180815    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180818    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180820    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:36.184017 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180823    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180826    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180828    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180831    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180834    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180837    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180839    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180842    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180845    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180847    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180850    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180852    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180855    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180857    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180860    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180863    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180865    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180868    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180870    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180873    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:36.184773 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180887    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180890    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180892    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180896    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180900    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180904    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180906    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180909    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180912    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180914    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180918    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180920    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180923    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180926    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180929    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180932    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180934    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180937    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180939    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:36.185539 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180942    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180946    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180951    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180954    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180957    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180960    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180963    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180966    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180968    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180971    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180973    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180976    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180978    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180981    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180984    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180986    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180989    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180991    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180994    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:36.186035 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.180997    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181071    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181118    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181126    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181130    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181134    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181137    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181141    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181146    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181149    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181152    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181155    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181158    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181162    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181165    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181168    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181172    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181175    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181178    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181183    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181185    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181188    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181191    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181195    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:43:36.186507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181198    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181201    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181205    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181208    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181211    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181215    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181218    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181221    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181224    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181228    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181233    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181236    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181239    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181242    2566 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181245    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181249    2566 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181253    2566 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181255    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181258    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181261    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181265    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181268    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181271    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181274    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181277    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:43:36.187092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181280    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181283    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181286    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181289    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181292    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181295    2566 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181298    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181301    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181304    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181307    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181310    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181313    2566 flags.go:64] FLAG: --help="false"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181316    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181319    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181322    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181325    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181328    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181333    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181336    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181339    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181341    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181344    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181347    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181350    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:43:36.187753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181353    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181356    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181358    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181362    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181364    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181367    2566 flags.go:64] FLAG: --lock-file=""
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181370    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181373    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181376    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181381    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181384    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181387    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181390    2566 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181393    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181396    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181399    2566 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181402    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181406    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181409    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181413    2566 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181416    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181419    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181421    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181424    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181428    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:43:36.188370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181433    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181436    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181443    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181446    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181449    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181452    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181455    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181460    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181463    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181466    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181469    2566 flags.go:64] FLAG: --port="10250"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181472    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181475    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f5b0be21e5ed124d"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181478    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181482    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181485    2566 flags.go:64] FLAG: --register-node="true"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181488    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181492    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181495    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181498    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181501    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181504    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181508    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181511    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181514    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:43:36.189035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181518    2566 flags.go:64] FLAG: --runonce="false"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181521    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181524    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181527    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181530    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181533    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181536    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181541    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181544    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181547    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181550    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181553    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181556    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181559    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181562    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181565    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181570    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181573    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181576    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181580    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181583    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181586    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181589    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181592    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181595    2566 flags.go:64] FLAG: --v="2"
Sep 16 18:43:36.189650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181599    2566 flags.go:64] FLAG: --version="false"
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181603    2566 flags.go:64] FLAG: --vmodule=""
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181607    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.181610    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181724    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181728    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181731    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181735    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181738    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181741    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181744    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181746    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181749    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181752    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181755    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181758    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181761    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181764    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181766    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181769    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181771    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:36.190289 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181774    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181777    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181779    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181782    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181785    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181788    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181790    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181793    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181795    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181798    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181800    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181803    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181808    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181811    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181814    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181817    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181820    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181823    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181827    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:36.190803 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181829    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181833    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181837    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181840    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181843    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181845    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181849    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181851    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181854    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181857    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181860    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181862    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181865    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181868    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181870    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181873    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181890    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181894    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181899    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:36.191282 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181903    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181906    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181909    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181911    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181914    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181917    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181920    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181923    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181926    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181928    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181931    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181934    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181937    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181942    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181945    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181948    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181951    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181954    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181956    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:36.191802 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181960    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181963    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181966    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181969    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181971    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181974    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181976    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181979    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181982    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181984    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181987    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181990    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181993    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181995    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.181999    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182001    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182004    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182007    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182009    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:36.192296 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182012    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182015    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182018    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182021    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182024    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.182027    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.182033    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.190413    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.190429    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.191759    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.191770    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191826    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191831    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191834    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:36.192770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191837    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191840    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191844    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191847    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191849    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191852    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191857    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191861    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191864    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191868    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191871    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191873    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191889    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191892    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191895    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191898    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191901    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191904    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191907    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:36.193170 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191910    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191913    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191915    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191918    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191921    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191931    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191936    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191939    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191942    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191945    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191948    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191951    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191953    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191956    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191959    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191962    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191965    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191968    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191971    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:36.193620 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191974    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191976    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191979    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191981    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191984    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191987    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191989    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191992    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191994    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.191997    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192000    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192002    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192005    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192007    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192010    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192013    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192016    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192018    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192021    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192024    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:36.194099 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192026    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192029    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192031    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192034    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192037    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192040    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192042    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192045    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192047    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192050    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192053    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192055    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192058    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192061    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192065    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192068    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192071    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192074    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192078    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192081    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:36.194587 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192084    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192087    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192089    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192092    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192094    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192097    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192099    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192102    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192104    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192107    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192109    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192112    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192115    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192118    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192120    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192123    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192126    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:36.195122 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192129    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.192134    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192256    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192263    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192266    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192269    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192273    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192276    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192278    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192281    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192284    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192287    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192290    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192292    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192296    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:43:36.195541 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192299    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192302    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192305    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192308    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192310    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192313    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192316    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192319    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192322    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192324    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192327    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192331    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192335    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192338    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192340    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192344    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192347    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192350    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192353    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:43:36.195984 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192355    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192358    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192361    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192363    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192366    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192368    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192371    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192374    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192376    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192379    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192382    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192386    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192389    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192393    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192396    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192399    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192402    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192405    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192408    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192410    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:43:36.196442 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192413    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192416    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192418    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192421    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192423    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192426    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192429    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192432    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192435    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192437    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192440    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192442    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192446    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192449    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192452    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192455    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192457    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192460    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192463    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:43:36.196935 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192465    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192468    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192470    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192473    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192475    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192478    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192480    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192484    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192487    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192489    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192492    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192495    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192497    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192499    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192502    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192505    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192507    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192510    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192513    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192515    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:43:36.197403 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192518    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192521    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192524    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192526    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192529    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192532    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192534    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:36.192537    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.192542    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.192656    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.196947    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:43:36.197937 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.197913    2566 server.go:1019] "Starting client certificate rotation"
Sep 16 18:43:36.198223 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.198017    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:36.198223 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.198051    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:43:36.200708 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.200695    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:36.203728 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.203700    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:43:36.212987 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.212970    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:43:36.216079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.216064    2566 log.go:25] "Validated CRI v1 image API"
Sep 16 18:43:36.216936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.216921    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:43:36.222483 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.222451    2566 fs.go:135] Filesystem UUIDs: map[1b29e0fe-12e0-4769-be6c-521160a59208:/dev/nvme0n1p3 6e6e31f1-1f1d-4486-852d-158b554aa61e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:43:36.222576 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.222484    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:43:36.228660 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.228550    2566 manager.go:216] Machine: {Timestamp:2026-09-16 18:43:36.228071658 +0000 UTC m=+0.339868290 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098496 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e86a2d7db2589420f3d80a9c2cff5 SystemUUID:ec2e86a2-d7db-2589-420f-3d80a9c2cff5 BootID:b329a8e9-4201-4721-b7b0-cdffcbccce09 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d3:3d:82:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d3:3d:82:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:ac:8f:dd:ee:ed Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:43:36.228660 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.228649    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:43:36.228785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.228759    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:43:36.231113 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.231088    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:43:36.231267 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.231116    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-190.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:43:36.231978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.231968    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:43:36.232019 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.231981    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:43:36.232019 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.231990    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:36.233718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.233705    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:43:36.235062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.235051    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:43:36.239696 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239684    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:36.239805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239795    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:43:36.239837 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239824    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:43:36.239837 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239833    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:43:36.239904 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239850    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:43:36.239904 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239859    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:43:36.239904 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.239869    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:43:36.240074 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.240058    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:36.240317 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.240305    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:36.240351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.240328    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:43:36.243421 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.243406    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:43:36.246550 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.246536    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:43:36.246610 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.246565    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:43:36.248234 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248212    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:43:36.248234 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248234    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248242    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248247    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248256    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248264    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248281    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248287    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248293    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248299    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248307    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:43:36.248353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.248314    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:43:36.249164 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.249153    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:43:36.249164 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.249165    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:43:36.250681 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.250655    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:43:36.250724 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.250655    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:43:36.252981 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.252968    2566 server.go:1268] "Started kubelet"
Sep 16 18:43:36.253110 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.253066    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:43:36.253163 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.253135    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:43:36.253685 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.253660    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:43:36.253749 ip-10-0-143-190 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:43:36.254665 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.254649    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:43:36.256179 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.255762    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:43:36.259996 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.259971    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:43:36.260342 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.260323    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-190.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:43:36.261106 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.260314    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-190.ec2.internal.18d5e15cf1670d4b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-190.ec2.internal,UID:ip-10-0-143-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-190.ec2.internal,},FirstTimestamp:2026-09-16 18:43:36.252943691 +0000 UTC m=+0.364740323,LastTimestamp:2026-09-16 18:43:36.252943691 +0000 UTC m=+0.364740323,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-190.ec2.internal,}"
Sep 16 18:43:36.262010 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.261995    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:43:36.262010 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.262004    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:36.262134 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.262116    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.262637 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.262615    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:43:36.262723 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.262655    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:43:36.262723 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.262673    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:43:36.263927 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.263908    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:43:36.263927 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.263925    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:43:36.264220 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264206    2566 factory.go:153] Registering CRI-O factory
Sep 16 18:43:36.264220 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264224    2566 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:43:36.264348 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264258    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mtwsw"
Sep 16 18:43:36.264348 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264305    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:43:36.264348 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264315    2566 factory.go:55] Registering systemd factory
Sep 16 18:43:36.264348 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264323    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:43:36.264348 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264339    2566 factory.go:103] Registering Raw factory
Sep 16 18:43:36.264536 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.264465    2566 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:43:36.265034 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.265021    2566 manager.go:349] Starting recovery of all containers
Sep 16 18:43:36.266596 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.266576    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-190.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 16 18:43:36.266680 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.266657    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 16 18:43:36.272027 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.271974    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mtwsw"
Sep 16 18:43:36.276655 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.276640    2566 manager.go:354] Recovery completed
Sep 16 18:43:36.280728 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.280712    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.283315 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283298    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.283389 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283329    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.283389 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283343    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.283806 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283793    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:43:36.283806 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283804    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:43:36.283923 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283818    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:43:36.283923 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.283864    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.284604 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.284588    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.284657 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.284614    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.284657 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.284625    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.285527 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.285451    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-190.ec2.internal.18d5e15cf336794d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-190.ec2.internal,UID:ip-10-0-143-190.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-190.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-190.ec2.internal,},FirstTimestamp:2026-09-16 18:43:36.283314509 +0000 UTC m=+0.395111147,LastTimestamp:2026-09-16 18:43:36.283314509 +0000 UTC m=+0.395111147,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-190.ec2.internal,}"
Sep 16 18:43:36.285988 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.285972    2566 policy_none.go:49] "None policy: Start"
Sep 16 18:43:36.285988 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.285988    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:43:36.286095 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.285999    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:43:36.288043 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.288027    2566 policy_none.go:47] "Start"
Sep 16 18:43:36.317633 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.317505    2566 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.317660    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.317672    2566 server.go:87] "Starting device plugin registration server"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.317936    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.317949    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.318041    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.318127    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.318135    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.318619    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:43:36.338785 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.318655    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.394821 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.394794    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:43:36.395944 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.395929    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:43:36.396000 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.395972    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:43:36.396000 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.395992    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:43:36.396083 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.396068    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:43:36.398044 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.398025    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:36.418923 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.418906    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.419661 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.419646    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.419744 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.419675    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.419744 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.419690    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.419744 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.419716    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.425899 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.425852    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.425899 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.425872    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-190.ec2.internal\": node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.451789 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.451770    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.496305 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.496286    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"]
Sep 16 18:43:36.496367 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.496348    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.497120 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.497106    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.497184 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.497133    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.497184 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.497142    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.499304 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499293    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.499425 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.499464 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499442    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.499976 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499959    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.500071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499983    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.500071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499992    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.500071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.499963    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.500071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.500017    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.500071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.500031    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.502188 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.502172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.502267 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.502202    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:43:36.502862 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.502847    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:43:36.502952 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.502867    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:43:36.502952 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.502892    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:43:36.515380 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.515359    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-190.ec2.internal\" not found" node="ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.519117 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.519101    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-190.ec2.internal\" not found" node="ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.552386 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.552368    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.565395 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.565377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60ddc793ed5e48a431d190fc8e8a6aa2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"60ddc793ed5e48a431d190fc8e8a6aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.565462 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.565398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60ddc793ed5e48a431d190fc8e8a6aa2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"60ddc793ed5e48a431d190fc8e8a6aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.565462 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.565418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eac4e7fa29b5d05ce71ef20ddfb25659-config\") pod \"kube-apiserver-proxy-ip-10-0-143-190.ec2.internal\" (UID: \"eac4e7fa29b5d05ce71ef20ddfb25659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.652449 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.652424    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.666142 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.666123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60ddc793ed5e48a431d190fc8e8a6aa2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"60ddc793ed5e48a431d190fc8e8a6aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.666199 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.666146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60ddc793ed5e48a431d190fc8e8a6aa2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"60ddc793ed5e48a431d190fc8e8a6aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.666199 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.666162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eac4e7fa29b5d05ce71ef20ddfb25659-config\") pod \"kube-apiserver-proxy-ip-10-0-143-190.ec2.internal\" (UID: \"eac4e7fa29b5d05ce71ef20ddfb25659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.666294 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.666217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eac4e7fa29b5d05ce71ef20ddfb25659-config\") pod \"kube-apiserver-proxy-ip-10-0-143-190.ec2.internal\" (UID: \"eac4e7fa29b5d05ce71ef20ddfb25659\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.666294 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.666227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/60ddc793ed5e48a431d190fc8e8a6aa2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"60ddc793ed5e48a431d190fc8e8a6aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.666294 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.666244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60ddc793ed5e48a431d190fc8e8a6aa2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal\" (UID: \"60ddc793ed5e48a431d190fc8e8a6aa2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.753055 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.752989    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.819405 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.819383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.823176 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:36.823157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:36.853152 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.853128    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:36.958048 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:36.958027    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:37.059715 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:37.059627    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:37.086510 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.086491    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:37.160669 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:37.160643    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-190.ec2.internal\" not found"
Sep 16 18:43:37.199786 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.199761    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:43:37.200434 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.199919    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:37.200434 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.199929    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:43:37.243808 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.243789    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:37.262102 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.262077    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:43:37.262188 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.262139    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:37.273102 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.273086    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:37.274167 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.274154    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal"
Sep 16 18:43:37.275311 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.275296    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:43:37.280611 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.280590    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:38:36 +0000 UTC" deadline="2028-02-29 00:38:00.791166337 +0000 UTC"
Sep 16 18:43:37.280662 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.280611    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12725h54m23.510557982s"
Sep 16 18:43:37.282842 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.282830    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:43:37.306022 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.306003    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxqwq"
Sep 16 18:43:37.315002 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.314961    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hxqwq"
Sep 16 18:43:37.551354 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.551327    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:37.664768 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:37.664740    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ddc793ed5e48a431d190fc8e8a6aa2.slice/crio-a13e82c4bc2b5eee89278399683223c391be7bc3b1679a67fe5e0be23f0e6ce2 WatchSource:0}: Error finding container a13e82c4bc2b5eee89278399683223c391be7bc3b1679a67fe5e0be23f0e6ce2: Status 404 returned error can't find the container with id a13e82c4bc2b5eee89278399683223c391be7bc3b1679a67fe5e0be23f0e6ce2
Sep 16 18:43:37.665196 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:37.665180    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeac4e7fa29b5d05ce71ef20ddfb25659.slice/crio-fa21afe7dc01c2a07e40b2747da7437aae10b8ae8f8a3ecc334d55ff78d5132b WatchSource:0}: Error finding container fa21afe7dc01c2a07e40b2747da7437aae10b8ae8f8a3ecc334d55ff78d5132b: Status 404 returned error can't find the container with id fa21afe7dc01c2a07e40b2747da7437aae10b8ae8f8a3ecc334d55ff78d5132b
Sep 16 18:43:37.671575 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:37.671552    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:43:38.228738 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.228705    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:38.241060 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.241039    2566 apiserver.go:52] "Watching apiserver"
Sep 16 18:43:38.259625 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.259600    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:43:38.260208 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.260179    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-xxlcv","openshift-network-operator/iptables-alerter-rmq9l","openshift-ovn-kubernetes/ovnkube-node-hqjqz","kube-system/global-pull-secret-syncer-42hjm","kube-system/konnectivity-agent-hbrg4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf","openshift-cluster-node-tuning-operator/tuned-9f7f9","openshift-image-registry/node-ca-bbdc7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal","openshift-multus/multus-additional-cni-plugins-57dch","openshift-network-diagnostics/network-check-target-cv2ng","kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal","openshift-dns/node-resolver-s22pq","openshift-multus/multus-jhpqj"]
Sep 16 18:43:38.263595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.263248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.263595 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.263332    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:38.265717 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.265685    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.270139 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.269423    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:43:38.270139 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.269435    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.270139 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.269716    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wxf5q\""
Sep 16 18:43:38.270139 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.269923    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.270398 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.270362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.273718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.273695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.273824 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.273776    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:38.274400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-host-slash\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.274470 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wf47\" (UniqueName: \"kubernetes.io/projected/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-kube-api-access-4wf47\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.274470 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.274567 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg8g5\" (UniqueName: \"kubernetes.io/projected/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-kube-api-access-bg8g5\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.274567 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-iptables-alerter-script\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.274712 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274689    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:43:38.274791 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.274774    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.275026 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.275010    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:43:38.275102 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.275037    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ql55z\""
Sep 16 18:43:38.275808 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.275789    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.278419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.278845    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.279219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.279238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.279795    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7lj9r\""
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.280300    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:43:38.281016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.280670    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:43:38.281722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.281564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.283136 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.282862    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.283478 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.283458    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-twbgl\""
Sep 16 18:43:38.284164 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.284136    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:43:38.284379 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.284360    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.285902 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.285866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.286543 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.286520    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.286675 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.286629    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ttpm5\""
Sep 16 18:43:38.286675 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.286526    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.288225 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.288205    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.288639 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.288617    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.288862 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.288832    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lljrs\""
Sep 16 18:43:38.289079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.289060    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:43:38.289165 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.289134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.291457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.291401    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:38.291529 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.291472    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:38.293383 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.293361    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:43:38.293631 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.293614    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fh4l5\""
Sep 16 18:43:38.293835 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.293819    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:43:38.294149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.294131    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.294344 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.294328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.294447 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.294427    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:43:38.294627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.294610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.296679 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.296658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.297393 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.297350    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:43:38.297551 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.297532    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:43:38.298133 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.298112    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rx2ts\""
Sep 16 18:43:38.299199 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.299179    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7ksft\""
Sep 16 18:43:38.299294 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.299249    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:43:38.317837 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.317806    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:37 +0000 UTC" deadline="2028-05-12 02:08:59.265803484 +0000 UTC"
Sep 16 18:43:38.317939 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.317837    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14479h25m20.947970677s"
Sep 16 18:43:38.364339 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.364318    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:43:38.375658 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-device-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375665    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-cni-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-run\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-lib-modules\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-system-cni-dir\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-conf-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.375759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-env-overrides\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysconfig\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-kubernetes\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ac3ad0-6e65-477d-8498-641c46de4e73-tmp-dir\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d7ff69b-8f46-4508-98be-1a0f2b166a20-cni-binary-copy\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375834    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-cni-multus\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-systemd-units\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczfh\" (UniqueName: \"kubernetes.io/projected/fae19076-36c9-4604-9d4a-af87f865ea97-kube-api-access-fczfh\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-modprobe-d\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.375950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmnz\" (UniqueName: \"kubernetes.io/projected/f7ac3ad0-6e65-477d-8498-641c46de4e73-kube-api-access-xxmnz\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38006a9b-3fb4-4888-8733-f4468724b1e5-dbus\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.376153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-tuned\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-etc-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-cni-bin\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-multus-certs\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-etc-kubernetes\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-host-slash\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f1787e6-b472-407d-8946-9d975b058d80-serviceca\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-netns\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-host-slash\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-cnibin\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-kubelet\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-systemd\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.376805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376553    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-cnibin\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376576    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfxj\" (UniqueName: \"kubernetes.io/projected/24a51272-3229-483c-a607-52ab5b7e9037-kube-api-access-hrfxj\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-run-netns\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-var-lib-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-cni-bin\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysctl-d\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ac3ad0-6e65-477d-8498-641c46de4e73-hosts-file\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-system-cni-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t597t\" (UniqueName: \"kubernetes.io/projected/d0507a55-a68a-430d-9352-63a68223ead2-kube-api-access-t597t\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdmdd\" (UniqueName: \"kubernetes.io/projected/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-kube-api-access-jdmdd\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-kubelet\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg8g5\" (UniqueName: \"kubernetes.io/projected/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-kube-api-access-bg8g5\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wf47\" (UniqueName: \"kubernetes.io/projected/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-kube-api-access-4wf47\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-socket-dir-parent\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-cni-netd\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.377632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-ovnkube-config\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.376982    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-sys\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/187276b7-26ce-4ea2-9709-359d01b580d9-agent-certs\") pod \"konnectivity-agent-hbrg4\" (UID: \"187276b7-26ce-4ea2-9709-359d01b580d9\") " pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-var-lib-kubelet\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-cni-binary-copy\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-k8s-cni-cncf-io\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-systemd\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-host\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377382    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/187276b7-26ce-4ea2-9709-359d01b580d9-konnectivity-ca\") pod \"konnectivity-agent-hbrg4\" (UID: \"187276b7-26ce-4ea2-9709-359d01b580d9\") " pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.377394    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-hostroot\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38006a9b-3fb4-4888-8733-f4468724b1e5-kubelet-config\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.377470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:38.877436486 +0000 UTC m=+2.989233132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377551    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysctl-conf\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.378338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-node-log\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fae19076-36c9-4604-9d4a-af87f865ea97-ovn-node-metrics-cert\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-iptables-alerter-script\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cct5n\" (UniqueName: \"kubernetes.io/projected/0f1787e6-b472-407d-8946-9d975b058d80-kube-api-access-cct5n\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-daemon-config\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.379068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-ovn\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379615 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-iptables-alerter-script\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.379720 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.377822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-ovnkube-script-lib\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379769 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-tmp\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.379810 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f1787e6-b472-407d-8946-9d975b058d80-host\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.379860 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk5jh\" (UniqueName: \"kubernetes.io/projected/1d7ff69b-8f46-4508-98be-1a0f2b166a20-kube-api-access-vk5jh\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.379899 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-slash\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379944 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379998 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-log-socket\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.379998 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.379979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.386642 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.386608    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:43:38.390881 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.390861    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wf47\" (UniqueName: \"kubernetes.io/projected/9bc1b081-d582-444f-8fe8-477a8cdbd4d9-kube-api-access-4wf47\") pod \"iptables-alerter-rmq9l\" (UID: \"9bc1b081-d582-444f-8fe8-477a8cdbd4d9\") " pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.391016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.390910    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg8g5\" (UniqueName: \"kubernetes.io/projected/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-kube-api-access-bg8g5\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.403195 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.403085    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" event={"ID":"60ddc793ed5e48a431d190fc8e8a6aa2","Type":"ContainerStarted","Data":"a13e82c4bc2b5eee89278399683223c391be7bc3b1679a67fe5e0be23f0e6ce2"}
Sep 16 18:43:38.404688 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.404664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal" event={"ID":"eac4e7fa29b5d05ce71ef20ddfb25659","Type":"ContainerStarted","Data":"fa21afe7dc01c2a07e40b2747da7437aae10b8ae8f8a3ecc334d55ff78d5132b"}
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczfh\" (UniqueName: \"kubernetes.io/projected/fae19076-36c9-4604-9d4a-af87f865ea97-kube-api-access-fczfh\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-modprobe-d\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmnz\" (UniqueName: \"kubernetes.io/projected/f7ac3ad0-6e65-477d-8498-641c46de4e73-kube-api-access-xxmnz\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.481385    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-modprobe-d\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.481470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:38.981449123 +0000 UTC m=+3.093245759 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38006a9b-3fb4-4888-8733-f4468724b1e5-dbus\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481532    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-tuned\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/38006a9b-3fb4-4888-8733-f4468724b1e5-dbus\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-etc-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481727    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-socket-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-cni-bin\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.481932 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-multus-certs\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-etc-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-etc-kubernetes\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-etc-kubernetes\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-cni-bin\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f1787e6-b472-407d-8946-9d975b058d80-serviceca\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-multus-certs\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-netns\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481927    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-netns\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-cnibin\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.481997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-kubelet\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-kubelet\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-systemd\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-cnibin\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.482945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-cnibin\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-sys-fs\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfxj\" (UniqueName: \"kubernetes.io/projected/24a51272-3229-483c-a607-52ab5b7e9037-kube-api-access-hrfxj\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482180    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-run-netns\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-var-lib-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482261    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-cni-bin\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysctl-d\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ac3ad0-6e65-477d-8498-641c46de4e73-hosts-file\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-system-cni-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-etc-selinux\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t597t\" (UniqueName: \"kubernetes.io/projected/d0507a55-a68a-430d-9352-63a68223ead2-kube-api-access-t597t\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdmdd\" (UniqueName: \"kubernetes.io/projected/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-kube-api-access-jdmdd\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-kubelet\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-socket-dir-parent\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-cni-netd\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0f1787e6-b472-407d-8946-9d975b058d80-serviceca\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.483706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-ovnkube-config\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482575    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-sys\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-socket-dir-parent\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-cnibin\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/187276b7-26ce-4ea2-9709-359d01b580d9-agent-certs\") pod \"konnectivity-agent-hbrg4\" (UID: \"187276b7-26ce-4ea2-9709-359d01b580d9\") " pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482664    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-run-netns\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-var-lib-kubelet\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-systemd\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-cni-binary-copy\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-var-lib-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482746    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-k8s-cni-cncf-io\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-systemd\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-host\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/187276b7-26ce-4ea2-9709-359d01b580d9-konnectivity-ca\") pod \"konnectivity-agent-hbrg4\" (UID: \"187276b7-26ce-4ea2-9709-359d01b580d9\") " pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-hostroot\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysctl-d\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38006a9b-3fb4-4888-8733-f4468724b1e5-kubelet-config\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.484436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-kubelet\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysctl-conf\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f7ac3ad0-6e65-477d-8498-641c46de4e73-hosts-file\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.482972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-node-log\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483030    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-node-log\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-system-cni-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-cni-netd\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fae19076-36c9-4604-9d4a-af87f865ea97-ovn-node-metrics-cert\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483111    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cct5n\" (UniqueName: \"kubernetes.io/projected/0f1787e6-b472-407d-8946-9d975b058d80-kube-api-access-cct5n\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-daemon-config\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-ovn\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-ovnkube-script-lib\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483275    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-tmp\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f1787e6-b472-407d-8946-9d975b058d80-host\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk5jh\" (UniqueName: \"kubernetes.io/projected/1d7ff69b-8f46-4508-98be-1a0f2b166a20-kube-api-access-vk5jh\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-slash\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.485368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-log-socket\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-device-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483487    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-ovnkube-config\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-cni-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483518    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-run\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-lib-modules\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-sys\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-system-cni-dir\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-lib-modules\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-conf-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-host\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-env-overrides\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysconfig\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/187276b7-26ce-4ea2-9709-359d01b580d9-konnectivity-ca\") pod \"konnectivity-agent-hbrg4\" (UID: \"187276b7-26ce-4ea2-9709-359d01b580d9\") " pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-kubernetes\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.486376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ac3ad0-6e65-477d-8498-641c46de4e73-tmp-dir\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-kubernetes\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.483945    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-hostroot\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/38006a9b-3fb4-4888-8733-f4468724b1e5-kubelet-config\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484158    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysctl-conf\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484369    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-var-lib-kubelet\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-run-k8s-cni-cncf-io\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-systemd\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-slash\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-openvswitch\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484611    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-log-socket\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-device-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484748    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-registration-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d7ff69b-8f46-4508-98be-1a0f2b166a20-cni-binary-copy\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-cni-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-cni-binary-copy\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.487238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485120    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d0507a55-a68a-430d-9352-63a68223ead2-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-cni-multus\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-systemd-units\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/24a51272-3229-483c-a607-52ab5b7e9037-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-host-var-lib-cni-multus\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-systemd-units\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7ac3ad0-6e65-477d-8498-641c46de4e73-tmp-dir\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.484674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0f1787e6-b472-407d-8946-9d975b058d80-host\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/24a51272-3229-483c-a607-52ab5b7e9037-system-cni-dir\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485392    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-sysconfig\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-run-ovn\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485462    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-etc-tuned\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-run\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-ovnkube-script-lib\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-run-ovn-kubernetes\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-daemon-config\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1d7ff69b-8f46-4508-98be-1a0f2b166a20-multus-conf-dir\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/187276b7-26ce-4ea2-9709-359d01b580d9-agent-certs\") pod \"konnectivity-agent-hbrg4\" (UID: \"187276b7-26ce-4ea2-9709-359d01b580d9\") " pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.487994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fae19076-36c9-4604-9d4a-af87f865ea97-host-cni-bin\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.488853 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.485920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1d7ff69b-8f46-4508-98be-1a0f2b166a20-cni-binary-copy\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.488853 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.486358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fae19076-36c9-4604-9d4a-af87f865ea97-env-overrides\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.488853 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.486986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fae19076-36c9-4604-9d4a-af87f865ea97-ovn-node-metrics-cert\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.488853 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.487721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-tmp\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.491322 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.491302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmnz\" (UniqueName: \"kubernetes.io/projected/f7ac3ad0-6e65-477d-8498-641c46de4e73-kube-api-access-xxmnz\") pod \"node-resolver-s22pq\" (UID: \"f7ac3ad0-6e65-477d-8498-641c46de4e73\") " pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.491468 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.491412    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:38.491581 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.491569    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:38.491670 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.491659    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:38.491822 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.491810    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:38.991792425 +0000 UTC m=+3.103589049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:38.492645 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.492618    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczfh\" (UniqueName: \"kubernetes.io/projected/fae19076-36c9-4604-9d4a-af87f865ea97-kube-api-access-fczfh\") pod \"ovnkube-node-hqjqz\" (UID: \"fae19076-36c9-4604-9d4a-af87f865ea97\") " pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.494497 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.494470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfxj\" (UniqueName: \"kubernetes.io/projected/24a51272-3229-483c-a607-52ab5b7e9037-kube-api-access-hrfxj\") pod \"multus-additional-cni-plugins-57dch\" (UID: \"24a51272-3229-483c-a607-52ab5b7e9037\") " pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.494871 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.494635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t597t\" (UniqueName: \"kubernetes.io/projected/d0507a55-a68a-430d-9352-63a68223ead2-kube-api-access-t597t\") pod \"aws-ebs-csi-driver-node-6ftdf\" (UID: \"d0507a55-a68a-430d-9352-63a68223ead2\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.494871 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.494659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk5jh\" (UniqueName: \"kubernetes.io/projected/1d7ff69b-8f46-4508-98be-1a0f2b166a20-kube-api-access-vk5jh\") pod \"multus-jhpqj\" (UID: \"1d7ff69b-8f46-4508-98be-1a0f2b166a20\") " pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.496068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.496047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdmdd\" (UniqueName: \"kubernetes.io/projected/ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd-kube-api-access-jdmdd\") pod \"tuned-9f7f9\" (UID: \"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd\") " pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.496215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.496141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cct5n\" (UniqueName: \"kubernetes.io/projected/0f1787e6-b472-407d-8946-9d975b058d80-kube-api-access-cct5n\") pod \"node-ca-bbdc7\" (UID: \"0f1787e6-b472-407d-8946-9d975b058d80\") " pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.588621 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.588586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rmq9l"
Sep 16 18:43:38.597410 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.597377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:43:38.601421 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:38.601392    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc1b081_d582_444f_8fe8_477a8cdbd4d9.slice/crio-5b00ac547ec1793c34fca973c9bb4f4086efee62af43f23bd62e8daa255d5d66 WatchSource:0}: Error finding container 5b00ac547ec1793c34fca973c9bb4f4086efee62af43f23bd62e8daa255d5d66: Status 404 returned error can't find the container with id 5b00ac547ec1793c34fca973c9bb4f4086efee62af43f23bd62e8daa255d5d66
Sep 16 18:43:38.607917 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.607877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:43:38.615223 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.615156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf"
Sep 16 18:43:38.622220 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.622195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9f7f9"
Sep 16 18:43:38.630972 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.630937    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bbdc7"
Sep 16 18:43:38.639450 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.639432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57dch"
Sep 16 18:43:38.649482 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.649464    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-s22pq"
Sep 16 18:43:38.656296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.656276    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jhpqj"
Sep 16 18:43:38.888453 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.888373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:38.888602 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.888525    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:38.888602 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.888593    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:39.888573219 +0000 UTC m=+4.000369840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:38.989269 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:38.989223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:38.989450 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.989426    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:38.989542 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:38.989530    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:39.989497672 +0000 UTC m=+4.101294311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:39.090488 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.090443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:39.090682 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.090634    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:39.090682 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.090661    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:39.090682 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.090672    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:39.090854 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.090765    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:40.090750075 +0000 UTC m=+4.202546700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:39.166341 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.166269    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:43:39.318403 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.318336    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:38:37 +0000 UTC" deadline="2028-04-26 01:04:18.063871263 +0000 UTC"
Sep 16 18:43:39.318403 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.318398    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14094h20m38.745477801s"
Sep 16 18:43:39.397231 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.397194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:39.397427 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.397403    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:39.407941 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.407904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmq9l" event={"ID":"9bc1b081-d582-444f-8fe8-477a8cdbd4d9","Type":"ContainerStarted","Data":"5b00ac547ec1793c34fca973c9bb4f4086efee62af43f23bd62e8daa255d5d66"}
Sep 16 18:43:39.816678 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.816641    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae19076_36c9_4604_9d4a_af87f865ea97.slice/crio-00d995432779dd5621a2ff7d1dfbd82373caacf7f83d848fe62c6a04e251798c WatchSource:0}: Error finding container 00d995432779dd5621a2ff7d1dfbd82373caacf7f83d848fe62c6a04e251798c: Status 404 returned error can't find the container with id 00d995432779dd5621a2ff7d1dfbd82373caacf7f83d848fe62c6a04e251798c
Sep 16 18:43:39.828218 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.828194    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1787e6_b472_407d_8946_9d975b058d80.slice/crio-35bbf1132113da0a84858b40ca5b0b16fb14e0e719c485a7d3aef2a03a1f4e6b WatchSource:0}: Error finding container 35bbf1132113da0a84858b40ca5b0b16fb14e0e719c485a7d3aef2a03a1f4e6b: Status 404 returned error can't find the container with id 35bbf1132113da0a84858b40ca5b0b16fb14e0e719c485a7d3aef2a03a1f4e6b
Sep 16 18:43:39.849961 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.849933    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a51272_3229_483c_a607_52ab5b7e9037.slice/crio-5cbcdc8c53d5e41921f856eb95d95c2120fb96b4035438ac0c2cecfc9e104060 WatchSource:0}: Error finding container 5cbcdc8c53d5e41921f856eb95d95c2120fb96b4035438ac0c2cecfc9e104060: Status 404 returned error can't find the container with id 5cbcdc8c53d5e41921f856eb95d95c2120fb96b4035438ac0c2cecfc9e104060
Sep 16 18:43:39.850748 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.850724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2711ac_8ee8_446b_88c5_e2f8a2ccaefd.slice/crio-60611299bd1debe634644f1cb6badd921f7e66262877810d764ae0ec41f71920 WatchSource:0}: Error finding container 60611299bd1debe634644f1cb6badd921f7e66262877810d764ae0ec41f71920: Status 404 returned error can't find the container with id 60611299bd1debe634644f1cb6badd921f7e66262877810d764ae0ec41f71920
Sep 16 18:43:39.852955 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.852921    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7ac3ad0_6e65_477d_8498_641c46de4e73.slice/crio-0e63a1c79f1115939b70bc1c6e3c86ce2d67d222d57ba64bd21807fbb215f5b0 WatchSource:0}: Error finding container 0e63a1c79f1115939b70bc1c6e3c86ce2d67d222d57ba64bd21807fbb215f5b0: Status 404 returned error can't find the container with id 0e63a1c79f1115939b70bc1c6e3c86ce2d67d222d57ba64bd21807fbb215f5b0
Sep 16 18:43:39.853673 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.853635    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d7ff69b_8f46_4508_98be_1a0f2b166a20.slice/crio-0f7db955fccaddc84cce5cc7146688ba543bf03c42e8000931fc431013a5dad9 WatchSource:0}: Error finding container 0f7db955fccaddc84cce5cc7146688ba543bf03c42e8000931fc431013a5dad9: Status 404 returned error can't find the container with id 0f7db955fccaddc84cce5cc7146688ba543bf03c42e8000931fc431013a5dad9
Sep 16 18:43:39.854795 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.854708    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0507a55_a68a_430d_9352_63a68223ead2.slice/crio-55b2b124190b4cefaa8993a3e46ca4dd12530fa622f9de1dc31efea0fd3a7423 WatchSource:0}: Error finding container 55b2b124190b4cefaa8993a3e46ca4dd12530fa622f9de1dc31efea0fd3a7423: Status 404 returned error can't find the container with id 55b2b124190b4cefaa8993a3e46ca4dd12530fa622f9de1dc31efea0fd3a7423
Sep 16 18:43:39.856263 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:43:39.856240    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187276b7_26ce_4ea2_9709_359d01b580d9.slice/crio-3360d0a91541a7627d343814d0dfb59ccfa390c66553e6aaf18575961a691c5d WatchSource:0}: Error finding container 3360d0a91541a7627d343814d0dfb59ccfa390c66553e6aaf18575961a691c5d: Status 404 returned error can't find the container with id 3360d0a91541a7627d343814d0dfb59ccfa390c66553e6aaf18575961a691c5d
Sep 16 18:43:39.897285 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.897252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:39.897434 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.897371    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:39.897434 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.897429    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:41.897413384 +0000 UTC m=+6.009210026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:39.997825 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:39.997782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:39.998027 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.997997    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:39.998097 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:39.998083    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:41.998061918 +0000 UTC m=+6.109858557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:40.098495 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.098397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:40.098656 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:40.098618    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:40.098656 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:40.098647    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:40.098756 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:40.098664    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:40.098756 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:40.098734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:42.098713241 +0000 UTC m=+6.210509869 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:40.396591 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.396499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:40.397075 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:40.396660    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:40.397075 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.396707    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:40.397075 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:40.396829    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:40.412024 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.411996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hbrg4" event={"ID":"187276b7-26ce-4ea2-9709-359d01b580d9","Type":"ContainerStarted","Data":"3360d0a91541a7627d343814d0dfb59ccfa390c66553e6aaf18575961a691c5d"}
Sep 16 18:43:40.413326 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.413297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf" event={"ID":"d0507a55-a68a-430d-9352-63a68223ead2","Type":"ContainerStarted","Data":"55b2b124190b4cefaa8993a3e46ca4dd12530fa622f9de1dc31efea0fd3a7423"}
Sep 16 18:43:40.414653 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.414627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerStarted","Data":"5cbcdc8c53d5e41921f856eb95d95c2120fb96b4035438ac0c2cecfc9e104060"}
Sep 16 18:43:40.416016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.415982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9f7f9" event={"ID":"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd","Type":"ContainerStarted","Data":"60611299bd1debe634644f1cb6badd921f7e66262877810d764ae0ec41f71920"}
Sep 16 18:43:40.417303 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.417270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bbdc7" event={"ID":"0f1787e6-b472-407d-8946-9d975b058d80","Type":"ContainerStarted","Data":"35bbf1132113da0a84858b40ca5b0b16fb14e0e719c485a7d3aef2a03a1f4e6b"}
Sep 16 18:43:40.418623 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.418590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jhpqj" event={"ID":"1d7ff69b-8f46-4508-98be-1a0f2b166a20","Type":"ContainerStarted","Data":"0f7db955fccaddc84cce5cc7146688ba543bf03c42e8000931fc431013a5dad9"}
Sep 16 18:43:40.419943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.419920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s22pq" event={"ID":"f7ac3ad0-6e65-477d-8498-641c46de4e73","Type":"ContainerStarted","Data":"0e63a1c79f1115939b70bc1c6e3c86ce2d67d222d57ba64bd21807fbb215f5b0"}
Sep 16 18:43:40.421375 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:40.421352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"00d995432779dd5621a2ff7d1dfbd82373caacf7f83d848fe62c6a04e251798c"}
Sep 16 18:43:41.397288 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.396953    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:41.398024 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:41.397373    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:41.435284 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.435248    2566 generic.go:356] "Generic (PLEG): container finished" podID="60ddc793ed5e48a431d190fc8e8a6aa2" containerID="fb449489728bf12658336a3233155e137384c68d7046d029038e02922f6bb1ad" exitCode=0
Sep 16 18:43:41.435885 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.435852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" event={"ID":"60ddc793ed5e48a431d190fc8e8a6aa2","Type":"ContainerDied","Data":"fb449489728bf12658336a3233155e137384c68d7046d029038e02922f6bb1ad"}
Sep 16 18:43:41.442846 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.442819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal" event={"ID":"eac4e7fa29b5d05ce71ef20ddfb25659","Type":"ContainerStarted","Data":"c895bd4c4bc65a12fb6e9a1ac07ad12c8f9abb47e87a2779b4e0d28ffbdb680f"}
Sep 16 18:43:41.448184 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.448155    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-s22pq" event={"ID":"f7ac3ad0-6e65-477d-8498-641c46de4e73","Type":"ContainerStarted","Data":"34eb714385d917e7dc41d72c46456fda33f18268ace52104ad66ee2d3ec1f60a"}
Sep 16 18:43:41.486985 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.486394    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-190.ec2.internal" podStartSLOduration=4.486370087 podStartE2EDuration="4.486370087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.469695641 +0000 UTC m=+5.581492286" watchObservedRunningTime="2026-09-16 18:43:41.486370087 +0000 UTC m=+5.598166732"
Sep 16 18:43:41.486985 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.486528    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-s22pq" podStartSLOduration=3.7314747219999997 podStartE2EDuration="4.486516537s" totalImagesPullingTime="755.041815ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:41.486308245 +0000 UTC m=+5.598104889" watchObservedRunningTime="2026-09-16 18:43:41.486516537 +0000 UTC m=+5.598313184"
Sep 16 18:43:41.914686 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:41.914295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:41.914686 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:41.914484    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:41.914686 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:41.914557    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:45.914538836 +0000 UTC m=+10.026335479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:42.016468 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.015804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:42.016468 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.016008    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:42.016468 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.016073    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:46.01605463 +0000 UTC m=+10.127851254 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:42.116531 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.116496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:42.116735 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.116673    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:42.116735 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.116693    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:42.116735 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.116707    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:42.116907 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.116764    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:46.116744218 +0000 UTC m=+10.228540841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:42.404718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.402590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:42.404718 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.403170    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:42.404718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.403660    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:42.404718 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:42.403829    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:42.461459 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.461372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" event={"ID":"60ddc793ed5e48a431d190fc8e8a6aa2","Type":"ContainerStarted","Data":"e260e80e7aa345d920bffd7b498728c92cdb3b7be916a7cf11a151421188ab97"}
Sep 16 18:43:42.468489 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.468456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rmq9l" event={"ID":"9bc1b081-d582-444f-8fe8-477a8cdbd4d9","Type":"ContainerStarted","Data":"1ab97253b0b54099b57acfa0e8d7855e3bf85d804f5ee850a849d5549fad1f23"}
Sep 16 18:43:42.482283 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:42.481942    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-190.ec2.internal" podStartSLOduration=5.481915865 podStartE2EDuration="5.481915865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:42.481884488 +0000 UTC m=+6.593681133" watchObservedRunningTime="2026-09-16 18:43:42.481915865 +0000 UTC m=+6.593712511"
Sep 16 18:43:43.397276 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:43.397232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:43.397464 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:43.397393    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:44.398060 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:44.398030    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:44.398685 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:44.398149    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:44.398685 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:44.398472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:44.398685 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:44.398570    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:45.396563 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:45.396339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:45.396723 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:45.396595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:45.952794 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:45.952753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:45.953217 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:45.952924    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:45.953217 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:45.952994    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:43:53.952974453 +0000 UTC m=+18.064771077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:46.053630 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:46.053286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:46.053630 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.053468    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:46.053630 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.053536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.053517782 +0000 UTC m=+18.165314404 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:46.153758 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:46.153676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:46.153952 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.153862    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:46.153952 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.153902    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:46.153952 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.153917    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:46.154097 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.153988    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:43:54.153968937 +0000 UTC m=+18.265765558 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:46.397838 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:46.397752    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:46.398000 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.397904    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:46.398000 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:46.397936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:46.398120 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:46.398062    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:47.397142 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:47.397108    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:47.397587 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:47.397322    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:48.396457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:48.396420    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:48.396642 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:48.396470    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:48.396642 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:48.396565    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:48.396913 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:48.396864    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:49.396585 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:49.396545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:49.397046 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:49.396678    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:50.397267 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:50.397236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:50.397724 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:50.397232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:50.397724 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:50.397515    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:50.398547 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:50.398053    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:51.396466 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:51.396425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:51.396644 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:51.396563    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:52.396897 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:52.396856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:52.397353 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:52.397005    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:52.397353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:52.397059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:52.397353 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:52.397152    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:53.396831 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:53.396744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:53.397070 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:53.396891    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:54.014778 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:54.014737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:54.014983 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.014855    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:54.014983 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.014931    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.014911177 +0000 UTC m=+34.126707797 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:43:54.115581 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:54.115541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:54.115744 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.115684    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:54.115784 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.115755    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.115737772 +0000 UTC m=+34.227534396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:43:54.215897 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:54.215844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:54.216071 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.216014    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:43:54.216071 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.216040    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:43:54.216071 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.216054    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:54.216229 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.216117    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.216098987 +0000 UTC m=+34.327895627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:43:54.396827 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:54.396750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:54.397008 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.396872    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:54.397008 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:54.396926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:54.397129 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:54.397015    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:55.396607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:55.396573    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:55.396794 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:55.396696    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:56.396828 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:56.396796    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:56.397326 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:56.396866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:56.397326 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:56.396901    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:56.397326 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:56.396961    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:57.396534 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.396232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:57.396534 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:57.396354    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:57.501910 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.501612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jhpqj" event={"ID":"1d7ff69b-8f46-4508-98be-1a0f2b166a20","Type":"ContainerStarted","Data":"7714e86d6a5b1e7710b6a56c24b130e379ae1734c8c35e0c6897a4776a3add73"}
Sep 16 18:43:57.503512 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.503471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hbrg4" event={"ID":"187276b7-26ce-4ea2-9709-359d01b580d9","Type":"ContainerStarted","Data":"fca58b91b5e42b476fabcad7692bc016cd68c8df25ea3d126b786c41663d00a1"}
Sep 16 18:43:57.505069 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.505048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf" event={"ID":"d0507a55-a68a-430d-9352-63a68223ead2","Type":"ContainerStarted","Data":"16f08f3fa0d60bffca7ec26e602978ad248bde3f31544386bb872b73b036dd47"}
Sep 16 18:43:57.506677 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.506655    2566 generic.go:356] "Generic (PLEG): container finished" podID="24a51272-3229-483c-a607-52ab5b7e9037" containerID="4ffbbc5b2ad8508b0fd6cf2bca7b51fe5d60e19303f1671244a1d5b9bc8ee3f4" exitCode=0
Sep 16 18:43:57.506793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.506768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerDied","Data":"4ffbbc5b2ad8508b0fd6cf2bca7b51fe5d60e19303f1671244a1d5b9bc8ee3f4"}
Sep 16 18:43:57.508336 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.508315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9f7f9" event={"ID":"ea2711ac-8ee8-446b-88c5-e2f8a2ccaefd","Type":"ContainerStarted","Data":"5451742df0fadd1eeb71e1aad1bd85164745e8661abfcb552ff983bcd53e5ec8"}
Sep 16 18:43:57.514940 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.514888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bbdc7" event={"ID":"0f1787e6-b472-407d-8946-9d975b058d80","Type":"ContainerStarted","Data":"7365a17be6312feff9b5cb7fac94a5b687493a18ceb18b2972e46d37d4cec3a2"}
Sep 16 18:43:57.518271 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.518238    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jhpqj" podStartSLOduration=3.138115691 podStartE2EDuration="20.518227934s" totalImagesPullingTime="17.380112243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:57.518103774 +0000 UTC m=+21.629900416" watchObservedRunningTime="2026-09-16 18:43:57.518227934 +0000 UTC m=+21.630024575"
Sep 16 18:43:57.518381 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.518363    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rmq9l" podStartSLOduration=18.531475264 podStartE2EDuration="20.518358801s" totalImagesPullingTime="1.986883537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:42.499731618 +0000 UTC m=+6.611528263" watchObservedRunningTime="2026-09-16 18:43:57.518358801 +0000 UTC m=+21.630155442"
Sep 16 18:43:57.532841 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.532803    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hbrg4" podStartSLOduration=7.411017816 podStartE2EDuration="20.532789707s" totalImagesPullingTime="13.121771891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:57.532668996 +0000 UTC m=+21.644465637" watchObservedRunningTime="2026-09-16 18:43:57.532789707 +0000 UTC m=+21.644586348"
Sep 16 18:43:57.551457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.551414    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9f7f9" podStartSLOduration=3.201697231 podStartE2EDuration="20.551398104s" totalImagesPullingTime="17.349700873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:57.550820114 +0000 UTC m=+21.662616756" watchObservedRunningTime="2026-09-16 18:43:57.551398104 +0000 UTC m=+21.663194746"
Sep 16 18:43:57.596659 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.596616    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bbdc7" podStartSLOduration=3.24443244 podStartE2EDuration="20.596598906s" totalImagesPullingTime="17.352166466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:57.596064837 +0000 UTC m=+21.707861479" watchObservedRunningTime="2026-09-16 18:43:57.596598906 +0000 UTC m=+21.708395548"
Sep 16 18:43:57.659799 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.659770    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:57.743682 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:57.743652    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:43:58.315315 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.315286    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:43:58.329758 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.329642    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:43:58.315310102Z","UUID":"8e6b4a0f-67c1-4225-abd7-ebf67039b291","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:43:58.331153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.331127    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:43:58.331257 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.331193    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:43:58.400326 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.400293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:43:58.400326 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.400311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:43:58.400607 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:58.400396    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:43:58.400607 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:58.400524    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:43:58.522464 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.522380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"dc3335f7d4952fd08d3673f9561b99062a67637555fe9010dff72791b4eac799"}
Sep 16 18:43:58.523149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.522471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"7b8fc592c1e00dd1fd2d5306453f7808b1ac73850c9e1a7f2794fdecafc25724"}
Sep 16 18:43:58.523149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.522523    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"8624cfd6637f3996485a72665bd7d495abbfb1bbe487443e28870282a4ecbfec"}
Sep 16 18:43:58.523149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.522570    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"a2160993385148ec269544c235cdc09dea4f42f16ef06b96b3cd886b77b32915"}
Sep 16 18:43:58.523149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.522619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"0bade4a79a4614d7359ab9d995c5e742747465b0e5293848b4c1778cd8191fe8"}
Sep 16 18:43:58.523149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.522666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"e3d5e09b42e73cb74a6ed38b43e9aa6ef0581545e8cdebaf4463366d705a6b00"}
Sep 16 18:43:58.525102 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:58.524946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf" event={"ID":"d0507a55-a68a-430d-9352-63a68223ead2","Type":"ContainerStarted","Data":"351e68491af862cd16238aa0112184c12cb031c55bd18dd382b966e5c221ebc4"}
Sep 16 18:43:59.396277 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:59.396201    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:43:59.396446 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:43:59.396353    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:43:59.530282 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:59.530242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf" event={"ID":"d0507a55-a68a-430d-9352-63a68223ead2","Type":"ContainerStarted","Data":"fe7d6cdc9bbd853fd624b72fc2b0300e70b64af8eb5a88601e2a8a4748f2e407"}
Sep 16 18:43:59.550559 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:43:59.550510    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6ftdf" podStartSLOduration=3.622664446 podStartE2EDuration="22.550494593s" totalImagesPullingTime="18.927830147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:43:59.550091564 +0000 UTC m=+23.661888206" watchObservedRunningTime="2026-09-16 18:43:59.550494593 +0000 UTC m=+23.662291233"
Sep 16 18:44:00.006923 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:00.006871    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:44:00.007510 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:00.007488    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:44:00.396607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:00.396567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:00.396785 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:00.396609    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:00.396785 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:00.396694    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:44:00.396785 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:00.396781    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:44:00.536964 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:00.536920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"9d3db186d7abd41a30813c0ca20d30da5197a9d65a73323ac473be1eb5457bc1"}
Sep 16 18:44:01.397134 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:01.397103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:01.397337 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:01.397243    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:44:02.396645 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.396559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:02.397113 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.396559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:02.397113 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:02.396676    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:44:02.397113 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:02.396743    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:44:02.544023 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.543667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" event={"ID":"fae19076-36c9-4604-9d4a-af87f865ea97","Type":"ContainerStarted","Data":"d62c138ef59ac3a79e990fa9a72644444e36532e85133b69dd3bf331603ff6fb"}
Sep 16 18:44:02.544023 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.543994    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:44:02.544222 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.544050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:44:02.545825 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.545800    2566 generic.go:356] "Generic (PLEG): container finished" podID="24a51272-3229-483c-a607-52ab5b7e9037" containerID="4e44d741086bd09262442b054f5888c063f464a021bb7951fcd4eb4372a098cb" exitCode=0
Sep 16 18:44:02.545962 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.545843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerDied","Data":"4e44d741086bd09262442b054f5888c063f464a021bb7951fcd4eb4372a098cb"}
Sep 16 18:44:02.559191 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.559170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:44:02.559494 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.559479    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:44:02.578372 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.578335    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz" podStartSLOduration=7.992626282 podStartE2EDuration="25.578320863s" totalImagesPullingTime="17.585694581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:02.577742501 +0000 UTC m=+26.689539152" watchObservedRunningTime="2026-09-16 18:44:02.578320863 +0000 UTC m=+26.690117503"
Sep 16 18:44:02.637215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.637185    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:02.714843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.714780    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:02.775886 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:02.775857    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:03.396970 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.396943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:03.397424 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:03.397042    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:44:03.550292 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.550261    2566 generic.go:356] "Generic (PLEG): container finished" podID="24a51272-3229-483c-a607-52ab5b7e9037" containerID="6762f873d1719cf6085c2823e79900cc3eb353b86df03ec2cfa2c13508842196" exitCode=0
Sep 16 18:44:03.550494 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.550350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerDied","Data":"6762f873d1719cf6085c2823e79900cc3eb353b86df03ec2cfa2c13508842196"}
Sep 16 18:44:03.550494 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.550415    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:44:03.560271 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.560249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:03.844840 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.844808    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-42hjm"]
Sep 16 18:44:03.845016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.844936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:03.845079 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:03.845034    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:44:03.848071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.848027    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cv2ng"]
Sep 16 18:44:03.848200 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.848129    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:03.848261 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:03.848218    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:44:03.848771 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.848747    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xxlcv"]
Sep 16 18:44:03.848873 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:03.848841    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:03.848984 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:03.848961    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:44:04.554457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:04.554378    2566 generic.go:356] "Generic (PLEG): container finished" podID="24a51272-3229-483c-a607-52ab5b7e9037" containerID="3e8bf7fdb740f372a10835ba31867cc2f7af0ea786382f3bd1e4bb3e5ed0bfbf" exitCode=0
Sep 16 18:44:04.555086 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:04.554453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerDied","Data":"3e8bf7fdb740f372a10835ba31867cc2f7af0ea786382f3bd1e4bb3e5ed0bfbf"}
Sep 16 18:44:04.555086 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:04.554501    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:44:04.639434 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:04.639406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:04.738134 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:04.738109    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:05.008046 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:05.008007    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:44:05.396960 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:05.396932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:05.397126 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:05.396933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:05.397218 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:05.396935    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:05.397218 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:05.397149    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:44:05.397218 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:05.397031    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:44:05.397351 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:05.397216    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:44:07.353151 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:07.353112    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:07.396940 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:07.396908    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:07.396940 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:07.396938    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:07.397140 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:07.396906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:07.397140 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:07.397048    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:44:07.397273 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:07.397133    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:44:07.397273 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:07.397227    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:44:07.446159 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:07.446114    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:07.987989 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:07.987939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.152978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.152940    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.240519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.240400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.341314 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.341276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.432975 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.432944    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:44:08.433461 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.433072    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 16 18:44:08.433652 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.433629    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hbrg4"
Sep 16 18:44:08.496943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.496856    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.557386 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.557356    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.823182 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.823116    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:08.931042 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:08.931011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:09.035936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.035900    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s22pq_f7ac3ad0-6e65-477d-8498-641c46de4e73/dns-node-resolver/0.log"
Sep 16 18:44:09.244163 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.244134    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:09.346609 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.346546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:09.396395 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.396376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:09.396517 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.396382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:09.396517 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:09.396467    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-cv2ng" podUID="0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd"
Sep 16 18:44:09.396594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.396382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:09.396594 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:09.396536    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-42hjm" podUID="38006a9b-3fb4-4888-8733-f4468724b1e5"
Sep 16 18:44:09.396672 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:09.396600    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxlcv" podUID="06f9ba4a-9f40-45f7-9ce4-1958a928ff0f"
Sep 16 18:44:09.569011 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.568979    2566 generic.go:356] "Generic (PLEG): container finished" podID="24a51272-3229-483c-a607-52ab5b7e9037" containerID="afa80e4df7d89f58e4fc44bc9860b101a25debba6c1dcf4e30d39be8410e2efc" exitCode=0
Sep 16 18:44:09.569405 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:09.569018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerDied","Data":"afa80e4df7d89f58e4fc44bc9860b101a25debba6c1dcf4e30d39be8410e2efc"}
Sep 16 18:44:10.026255 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.026168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:10.026420 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.026285    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:10.026420 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.026336    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs podName:06f9ba4a-9f40-45f7-9ce4-1958a928ff0f nodeName:}" failed. No retries permitted until 2026-09-16 18:44:42.026323429 +0000 UTC m=+66.138120048 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs") pod "network-metrics-daemon-xxlcv" (UID: "06f9ba4a-9f40-45f7-9ce4-1958a928ff0f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:10.097044 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.097021    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-190.ec2.internal" event="NodeReady"
Sep 16 18:44:10.097170 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.097123    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:44:10.127068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.127040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:10.127169 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.127161    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:10.127217 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.127208    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret podName:38006a9b-3fb4-4888-8733-f4468724b1e5 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:42.127194749 +0000 UTC m=+66.238991368 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret") pod "global-pull-secret-syncer-42hjm" (UID: "38006a9b-3fb4-4888-8733-f4468724b1e5") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:10.138268 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.138241    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5969f9b54c-zrghd"]
Sep 16 18:44:10.156124 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.156102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.158901 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.158863    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-94j8c"]
Sep 16 18:44:10.160846 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.160789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:44:10.160977 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.160860    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:44:10.160977 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.160899    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4tghj\""
Sep 16 18:44:10.160977 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.160930    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:44:10.168117 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.167951    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:44:10.172300 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.172279    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xqjz5"]
Sep 16 18:44:10.172399 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.172384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.176400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.176384    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:44:10.176498 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.176414    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xrh57\""
Sep 16 18:44:10.176675 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.176633    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:44:10.190708 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.190689    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5969f9b54c-zrghd"]
Sep 16 18:44:10.190796 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.190715    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94j8c"]
Sep 16 18:44:10.190796 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.190739    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4r4ns"]
Sep 16 18:44:10.190796 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.190790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.193575 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.193559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:44:10.193656 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.193574    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6r5qx\""
Sep 16 18:44:10.193703 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.193678    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:44:10.193908 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.193868    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:44:10.194014 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.193995    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:44:10.210995 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.210978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xqjz5"]
Sep 16 18:44:10.211082 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.211009    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4r4ns"]
Sep 16 18:44:10.211082 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.211068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.215273 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.215242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:44:10.215376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.215302    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:44:10.215376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.215366    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ktvzd\""
Sep 16 18:44:10.215847 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.215833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:44:10.228229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:10.228308 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-installation-pull-secrets\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228308 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-bound-sa-token\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228308 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdp9x\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-kube-api-access-wdp9x\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228308 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228430 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.228363    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:10.228430 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.228384    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:10.228430 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.228395    2566 projected.go:196] Error preparing data for projected volume kube-api-access-m4j4z for pod openshift-network-diagnostics/network-check-target-cv2ng: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:10.228519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-certificates\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228519 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.228442    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z podName:0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd nodeName:}" failed. No retries permitted until 2026-09-16 18:44:42.228426428 +0000 UTC m=+66.340223050 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4j4z" (UniqueName: "kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z") pod "network-check-target-cv2ng" (UID: "0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:10.228519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-trusted-ca\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-image-registry-private-configuration\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.228640 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.228531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-ca-trust-extracted\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.237068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.237053    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbdc7_0f1787e6-b472-407d-8946-9d975b058d80/node-ca/0.log"
Sep 16 18:44:10.312850 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.312792    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:10.329759 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-installation-pull-secrets\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.329829 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.329829 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-bound-sa-token\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.329829 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdp9x\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-kube-api-access-wdp9x\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.329993 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.329993 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.329993 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.329993 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmlzw\" (UniqueName: \"kubernetes.io/projected/6f124bba-8be5-4999-86ba-40e295e86078-kube-api-access-qmlzw\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.329988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6-cert\") pod \"ingress-canary-4r4ns\" (UID: \"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6\") " pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s2dm\" (UniqueName: \"kubernetes.io/projected/d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6-kube-api-access-6s2dm\") pod \"ingress-canary-4r4ns\" (UID: \"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6\") " pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.330031    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.330057    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5969f9b54c-zrghd: secret "image-registry-tls" not found
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-certificates\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f124bba-8be5-4999-86ba-40e295e86078-crio-socket\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.330120    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls podName:a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.830098885 +0000 UTC m=+34.941895529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls") pod "image-registry-5969f9b54c-zrghd" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b") : secret "image-registry-tls" not found
Sep 16 18:44:10.330186 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-trusted-ca\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.330538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a79dc4-f304-4b8d-a892-f3836095cfe0-tmp-dir\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.330538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-image-registry-private-configuration\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.330538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-ca-trust-extracted\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.330538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a79dc4-f304-4b8d-a892-f3836095cfe0-config-volume\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.330538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330334    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrsgk\" (UniqueName: \"kubernetes.io/projected/51a79dc4-f304-4b8d-a892-f3836095cfe0-kube-api-access-xrsgk\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.330538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f124bba-8be5-4999-86ba-40e295e86078-data-volume\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.330795 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-ca-trust-extracted\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.330795 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-certificates\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.331015 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.330991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-trusted-ca\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.333816 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.333799    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-installation-pull-secrets\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.333816 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.333809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-image-registry-private-configuration\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.338528 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.338503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-bound-sa-token\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.338620 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.338604    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdp9x\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-kube-api-access-wdp9x\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.424597 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.424570    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-djm56"]
Sep 16 18:44:10.430705 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.430684    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.430792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.430723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.430792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.430741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.430870 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.430806    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:10.430870 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.430833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmlzw\" (UniqueName: \"kubernetes.io/projected/6f124bba-8be5-4999-86ba-40e295e86078-kube-api-access-qmlzw\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.430870 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.430839    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:10.430870 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.430865    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.430849    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls podName:51a79dc4-f304-4b8d-a892-f3836095cfe0 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.930838397 +0000 UTC m=+35.042635016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls") pod "dns-default-94j8c" (UID: "51a79dc4-f304-4b8d-a892-f3836095cfe0") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.430950    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.930932952 +0000 UTC m=+35.042729575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.430963    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:10.930956496 +0000 UTC m=+35.042753115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.430974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6-cert\") pod \"ingress-canary-4r4ns\" (UID: \"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6\") " pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.430992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s2dm\" (UniqueName: \"kubernetes.io/projected/d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6-kube-api-access-6s2dm\") pod \"ingress-canary-4r4ns\" (UID: \"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6\") " pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f124bba-8be5-4999-86ba-40e295e86078-crio-socket\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.431069 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a79dc4-f304-4b8d-a892-f3836095cfe0-tmp-dir\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.431355 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a79dc4-f304-4b8d-a892-f3836095cfe0-config-volume\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.431355 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsgk\" (UniqueName: \"kubernetes.io/projected/51a79dc4-f304-4b8d-a892-f3836095cfe0-kube-api-access-xrsgk\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.431355 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f124bba-8be5-4999-86ba-40e295e86078-data-volume\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.431355 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6f124bba-8be5-4999-86ba-40e295e86078-crio-socket\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.431482 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6f124bba-8be5-4999-86ba-40e295e86078-data-volume\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.431515 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51a79dc4-f304-4b8d-a892-f3836095cfe0-tmp-dir\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.431592 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.431575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51a79dc4-f304-4b8d-a892-f3836095cfe0-config-volume\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.433066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.433051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6-cert\") pod \"ingress-canary-4r4ns\" (UID: \"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6\") " pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.445042 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.445014    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmlzw\" (UniqueName: \"kubernetes.io/projected/6f124bba-8be5-4999-86ba-40e295e86078-kube-api-access-qmlzw\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.445953 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.445932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrsgk\" (UniqueName: \"kubernetes.io/projected/51a79dc4-f304-4b8d-a892-f3836095cfe0-kube-api-access-xrsgk\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.446349 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.446334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s2dm\" (UniqueName: \"kubernetes.io/projected/d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6-kube-api-access-6s2dm\") pod \"ingress-canary-4r4ns\" (UID: \"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6\") " pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.447512 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.447492    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-djm56"]
Sep 16 18:44:10.447607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.447581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.450079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.450059    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 16 18:44:10.450191 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.450165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 16 18:44:10.450290 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.450190    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9wh8z\""
Sep 16 18:44:10.450290 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.450264    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 16 18:44:10.450290 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.450268    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 16 18:44:10.468148 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.468125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:10.519430 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.519410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4r4ns"
Sep 16 18:44:10.530974 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.530947    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:10.531922 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.531904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e34b6f7-06ae-4785-a126-4213f8146087-signing-cabundle\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.531978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.531962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e34b6f7-06ae-4785-a126-4213f8146087-signing-key\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.532052 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.532009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mpt\" (UniqueName: \"kubernetes.io/projected/5e34b6f7-06ae-4785-a126-4213f8146087-kube-api-access-27mpt\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.574774 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.574704    2566 generic.go:356] "Generic (PLEG): container finished" podID="24a51272-3229-483c-a607-52ab5b7e9037" containerID="f713f9ee08e906414451d72ec4e4793e8fead6d062183f0598d367646d706d8d" exitCode=0
Sep 16 18:44:10.574774 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.574754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerDied","Data":"f713f9ee08e906414451d72ec4e4793e8fead6d062183f0598d367646d706d8d"}
Sep 16 18:44:10.633202 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.632964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e34b6f7-06ae-4785-a126-4213f8146087-signing-cabundle\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.633317 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.633241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e34b6f7-06ae-4785-a126-4213f8146087-signing-key\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.633317 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.633284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mpt\" (UniqueName: \"kubernetes.io/projected/5e34b6f7-06ae-4785-a126-4213f8146087-kube-api-access-27mpt\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.633619 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.633594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/5e34b6f7-06ae-4785-a126-4213f8146087-signing-cabundle\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.636594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.636569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/5e34b6f7-06ae-4785-a126-4213f8146087-signing-key\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.641303 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.641282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mpt\" (UniqueName: \"kubernetes.io/projected/5e34b6f7-06ae-4785-a126-4213f8146087-kube-api-access-27mpt\") pod \"service-ca-568869fdc5-djm56\" (UID: \"5e34b6f7-06ae-4785-a126-4213f8146087\") " pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.756492 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.756468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-djm56"
Sep 16 18:44:10.830142 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.830086    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4r4ns"]
Sep 16 18:44:10.833235 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:10.833207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c0e28f_7c81_4d9b_8ca4_0d3fd594ddd6.slice/crio-a9c59daa5243bec24010ee2694bc76af116ba689b559f8beaf474f111298eefd WatchSource:0}: Error finding container a9c59daa5243bec24010ee2694bc76af116ba689b559f8beaf474f111298eefd: Status 404 returned error can't find the container with id a9c59daa5243bec24010ee2694bc76af116ba689b559f8beaf474f111298eefd
Sep 16 18:44:10.834683 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.834661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:10.834818 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.834802    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:10.834873 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.834819    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5969f9b54c-zrghd: secret "image-registry-tls" not found
Sep 16 18:44:10.834873 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.834867    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls podName:a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.834853883 +0000 UTC m=+35.946650501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls") pod "image-registry-5969f9b54c-zrghd" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b") : secret "image-registry-tls" not found
Sep 16 18:44:10.936053 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.936026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.936202 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.936092    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:10.936202 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:10.936113    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:10.936314 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.936207    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:10.936314 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.936214    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:10.936314 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.936238    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:10.936314 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.936246    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls podName:51a79dc4-f304-4b8d-a892-f3836095cfe0 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.93623442 +0000 UTC m=+36.048031038 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls") pod "dns-default-94j8c" (UID: "51a79dc4-f304-4b8d-a892-f3836095cfe0") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:10.936314 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.936314    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.936296172 +0000 UTC m=+36.048092806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:10.936500 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:10.936331    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:11.936321914 +0000 UTC m=+36.048118532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:11.051407 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.051277    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-djm56"]
Sep 16 18:44:11.053813 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:11.053790    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e34b6f7_06ae_4785_a126_4213f8146087.slice/crio-b20cddfcf6f71ea31c3643fcb9d18bda8fd2ce58b6443f0f683ba04c999e8814 WatchSource:0}: Error finding container b20cddfcf6f71ea31c3643fcb9d18bda8fd2ce58b6443f0f683ba04c999e8814: Status 404 returned error can't find the container with id b20cddfcf6f71ea31c3643fcb9d18bda8fd2ce58b6443f0f683ba04c999e8814
Sep 16 18:44:11.396380 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.396338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:11.396626 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.396454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:11.396675 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.396636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:11.399147 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.399121    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:11.399453 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.399327    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:11.399453 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.399398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mz62r\""
Sep 16 18:44:11.399645 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.399539    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:11.400545 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.400516    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fq6l6\""
Sep 16 18:44:11.400545 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.400527    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:11.410584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.410559    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:11.579713 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.579677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-djm56" event={"ID":"5e34b6f7-06ae-4785-a126-4213f8146087","Type":"ContainerStarted","Data":"b20cddfcf6f71ea31c3643fcb9d18bda8fd2ce58b6443f0f683ba04c999e8814"}
Sep 16 18:44:11.581989 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.581959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4r4ns" event={"ID":"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6","Type":"ContainerStarted","Data":"a9c59daa5243bec24010ee2694bc76af116ba689b559f8beaf474f111298eefd"}
Sep 16 18:44:11.586737 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.586713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57dch" event={"ID":"24a51272-3229-483c-a607-52ab5b7e9037","Type":"ContainerStarted","Data":"ef3382b65aae646bfe07b0e517253cf7f02bac7c7c8498d20f3174dec2f693b3"}
Sep 16 18:44:11.609008 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.608955    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-57dch" podStartSLOduration=7.135850338 podStartE2EDuration="34.60893844s" totalImagesPullingTime="27.473088102s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:11.607133155 +0000 UTC m=+35.718929847" watchObservedRunningTime="2026-09-16 18:44:11.60893844 +0000 UTC m=+35.720735082"
Sep 16 18:44:11.843716 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.843591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:11.843967 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.843805    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:11.843967 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.843830    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5969f9b54c-zrghd: secret "image-registry-tls" not found
Sep 16 18:44:11.843967 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.843911    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls podName:a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.843874628 +0000 UTC m=+37.955671249 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls") pod "image-registry-5969f9b54c-zrghd" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b") : secret "image-registry-tls" not found
Sep 16 18:44:11.944922 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.944871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:11.945107 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.944960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:11.945107 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:11.944990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:11.945107 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.945026    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:11.945282 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.945108    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:11.945282 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.945117    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:11.945282 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.945119    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.945097434 +0000 UTC m=+38.056894067 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:11.945282 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.945159    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls podName:51a79dc4-f304-4b8d-a892-f3836095cfe0 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.945147711 +0000 UTC m=+38.056944330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls") pod "dns-default-94j8c" (UID: "51a79dc4-f304-4b8d-a892-f3836095cfe0") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:11.945282 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:11.945175    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:13.945168242 +0000 UTC m=+38.056964860 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:13.861521 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:13.861417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:13.861875 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.861571    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:13.861875 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.861593    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5969f9b54c-zrghd: secret "image-registry-tls" not found
Sep 16 18:44:13.861875 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.861651    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls podName:a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:17.861636072 +0000 UTC m=+41.973432692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls") pod "image-registry-5969f9b54c-zrghd" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b") : secret "image-registry-tls" not found
Sep 16 18:44:13.962745 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:13.962713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:13.962915 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:13.962757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:13.962915 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:13.962776    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:13.962915 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.962842    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:13.962915 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.962860    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:13.963120 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.962920    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:17.962902436 +0000 UTC m=+42.074699055 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:13.963120 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.962946    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:13.963120 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.962996    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:17.962971578 +0000 UTC m=+42.074768212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:13.963120 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:13.963031    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls podName:51a79dc4-f304-4b8d-a892-f3836095cfe0 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:17.963020151 +0000 UTC m=+42.074816777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls") pod "dns-default-94j8c" (UID: "51a79dc4-f304-4b8d-a892-f3836095cfe0") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:14.596070 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:14.595758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-djm56" event={"ID":"5e34b6f7-06ae-4785-a126-4213f8146087","Type":"ContainerStarted","Data":"cffc045cb8c061e6161fe2a7877e70dfa2ba4e1967746b6b370e8c4a44ee3da1"}
Sep 16 18:44:14.597333 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:14.597312    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4r4ns" event={"ID":"d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6","Type":"ContainerStarted","Data":"430b7636816a79610d424444f5694cc5c982219838bc0db078022e5d5c74da32"}
Sep 16 18:44:14.612621 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:14.612581    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-djm56" podStartSLOduration=1.5080886740000001 podStartE2EDuration="4.612569582s" totalImagesPullingTime="3.104480908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:14.611524462 +0000 UTC m=+38.723321102" watchObservedRunningTime="2026-09-16 18:44:14.612569582 +0000 UTC m=+38.724366217"
Sep 16 18:44:14.628629 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:14.628582    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4r4ns" podStartSLOduration=1.309177157 podStartE2EDuration="4.62856596s" totalImagesPullingTime="3.319388803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:14.627542982 +0000 UTC m=+38.739339638" watchObservedRunningTime="2026-09-16 18:44:14.62856596 +0000 UTC m=+38.740362600"
Sep 16 18:44:15.420087 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:15.420056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:15.621936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:15.621901    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:15.789206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:15.789079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:15.963561 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:15.963532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:16.067505 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:16.067412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:16.167044 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:16.167008    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:16.514605 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:16.514522    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:17.119036 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.119000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:17.507161 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.507079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:17.710900 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.710845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:17.894028 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.893952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:17.894195 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.894120    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:44:17.894195 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.894140    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5969f9b54c-zrghd: secret "image-registry-tls" not found
Sep 16 18:44:17.894303 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.894207    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls podName:a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.894186178 +0000 UTC m=+50.005982812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls") pod "image-registry-5969f9b54c-zrghd" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b") : secret "image-registry-tls" not found
Sep 16 18:44:17.994896 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.994856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:17.995046 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.994927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:17.995046 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:17.994949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:17.995046 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.994977    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:44:17.995046 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.995042    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.99502616 +0000 UTC m=+50.106822779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : configmap "kube-rbac-proxy" not found
Sep 16 18:44:17.995183 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.995081    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:44:17.995183 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.995127    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls podName:51a79dc4-f304-4b8d-a892-f3836095cfe0 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.995116168 +0000 UTC m=+50.106912786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls") pod "dns-default-94j8c" (UID: "51a79dc4-f304-4b8d-a892-f3836095cfe0") : secret "dns-default-metrics-tls" not found
Sep 16 18:44:17.995183 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.995081    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:17.995183 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:17.995160    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls podName:6f124bba-8be5-4999-86ba-40e295e86078 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:25.995150593 +0000 UTC m=+50.106947212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xqjz5" (UID: "6f124bba-8be5-4999-86ba-40e295e86078") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:44:18.319710 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:18.319613    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:18.911253 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:18.911224    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.119938 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:19.119854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.313624 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:19.313534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.523197 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:19.523162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.728734 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:19.728682    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:19.829930 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:19.829855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.027477 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:20.027403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.321456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:20.321370    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.388378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:20.388342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:20.918181 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:20.918149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.116090 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.116056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.315152 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.315056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.414159 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.414125    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.524602 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.524572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.645922 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.645898    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.743740 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.743711    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:21.996550 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:21.996463    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:22.321564 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:22.321468    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:22.718693 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:22.718656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:23.096566 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:23.096534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:23.254684 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:23.254601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:23.536009 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:23.535929    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:23.735796 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:23.735762    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:24.115554 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:24.115520    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:24.913789 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:24.913734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:25.122137 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:25.122104    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:25.315414 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:25.315322    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:25.711157 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:25.711118    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:25.956250 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:25.956217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:25.959643 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:25.959621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"image-registry-5969f9b54c-zrghd\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") " pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:26.057329 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.057252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:26.057329 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.057291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:26.057526 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.057403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:26.057905 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.057862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6f124bba-8be5-4999-86ba-40e295e86078-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:26.059489 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.059472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6f124bba-8be5-4999-86ba-40e295e86078-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xqjz5\" (UID: \"6f124bba-8be5-4999-86ba-40e295e86078\") " pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:26.059579 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.059558    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51a79dc4-f304-4b8d-a892-f3836095cfe0-metrics-tls\") pod \"dns-default-94j8c\" (UID: \"51a79dc4-f304-4b8d-a892-f3836095cfe0\") " pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:26.070510 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.070482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:26.081323 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.081302    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:26.099574 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.099549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xqjz5"
Sep 16 18:44:26.121360 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.121328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.320702 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.320476    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:26.358828 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.358734    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5969f9b54c-zrghd"]
Sep 16 18:44:26.363768 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:26.363732    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6ed46c3_8dce_4fe8_a9cf_0406d4440b3b.slice/crio-4e953c1539d0d8f14999ff9a4cf44587be9059c1426fe78649e77ff1575fbc2f WatchSource:0}: Error finding container 4e953c1539d0d8f14999ff9a4cf44587be9059c1426fe78649e77ff1575fbc2f: Status 404 returned error can't find the container with id 4e953c1539d0d8f14999ff9a4cf44587be9059c1426fe78649e77ff1575fbc2f
Sep 16 18:44:26.374983 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.374953    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-94j8c"]
Sep 16 18:44:26.378514 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:26.378495    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51a79dc4_f304_4b8d_a892_f3836095cfe0.slice/crio-62b5acac57bdf634d1eb08cb147d72338e2fd7e0b53304fa5f3e9d0475c1e20d WatchSource:0}: Error finding container 62b5acac57bdf634d1eb08cb147d72338e2fd7e0b53304fa5f3e9d0475c1e20d: Status 404 returned error can't find the container with id 62b5acac57bdf634d1eb08cb147d72338e2fd7e0b53304fa5f3e9d0475c1e20d
Sep 16 18:44:26.393190 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.393168    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xqjz5"]
Sep 16 18:44:26.395925 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:26.395872    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f124bba_8be5_4999_86ba_40e295e86078.slice/crio-521d5a7f6336f75c596e7a23d2b3d5abb8bb2dd1a9f5b0d59a42c9e4dee55c25 WatchSource:0}: Error finding container 521d5a7f6336f75c596e7a23d2b3d5abb8bb2dd1a9f5b0d59a42c9e4dee55c25: Status 404 returned error can't find the container with id 521d5a7f6336f75c596e7a23d2b3d5abb8bb2dd1a9f5b0d59a42c9e4dee55c25
Sep 16 18:44:26.630701 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.630619    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94j8c" event={"ID":"51a79dc4-f304-4b8d-a892-f3836095cfe0","Type":"ContainerStarted","Data":"62b5acac57bdf634d1eb08cb147d72338e2fd7e0b53304fa5f3e9d0475c1e20d"}
Sep 16 18:44:26.632141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.632112    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xqjz5" event={"ID":"6f124bba-8be5-4999-86ba-40e295e86078","Type":"ContainerStarted","Data":"f9c8429de94bcbd31f701646dea17f1eca6154b02bfeb42e434794b2b8407ca1"}
Sep 16 18:44:26.632265 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.632157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xqjz5" event={"ID":"6f124bba-8be5-4999-86ba-40e295e86078","Type":"ContainerStarted","Data":"521d5a7f6336f75c596e7a23d2b3d5abb8bb2dd1a9f5b0d59a42c9e4dee55c25"}
Sep 16 18:44:26.633648 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.633613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" event={"ID":"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b","Type":"ContainerStarted","Data":"c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867"}
Sep 16 18:44:26.633768 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.633669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" event={"ID":"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b","Type":"ContainerStarted","Data":"4e953c1539d0d8f14999ff9a4cf44587be9059c1426fe78649e77ff1575fbc2f"}
Sep 16 18:44:26.633768 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.633737    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:26.655500 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:26.655458    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" podStartSLOduration=61.655444315 podStartE2EDuration="1m1.655444315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:26.654072693 +0000 UTC m=+50.765869336" watchObservedRunningTime="2026-09-16 18:44:26.655444315 +0000 UTC m=+50.767240947"
Sep 16 18:44:27.640116 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:27.640007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xqjz5" event={"ID":"6f124bba-8be5-4999-86ba-40e295e86078","Type":"ContainerStarted","Data":"6dc80f028f2dd58b41ca5d76c78874768d98ee21e3ceceeb11fb6641f1b2288f"}
Sep 16 18:44:28.076293 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.076209    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.175654 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.175613    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.275298 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.275261    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.647435 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.647402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94j8c" event={"ID":"51a79dc4-f304-4b8d-a892-f3836095cfe0","Type":"ContainerStarted","Data":"95a47986f856f6815c039515084ac6f5bb21170f17345a911976435cff56296c"}
Sep 16 18:44:28.647889 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.647458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-94j8c" event={"ID":"51a79dc4-f304-4b8d-a892-f3836095cfe0","Type":"ContainerStarted","Data":"86e637720d21e8c3035cc6c84be1244a29fb8cb3444664aaa7ba7954924857eb"}
Sep 16 18:44:28.647889 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.647510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:28.649711 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.649687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xqjz5" event={"ID":"6f124bba-8be5-4999-86ba-40e295e86078","Type":"ContainerStarted","Data":"2c7fdbad2231cc0425bdaa231bdc2dcda27305102789a1e3fea33aef9e718956"}
Sep 16 18:44:28.666596 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.666552    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-94j8c" podStartSLOduration=16.729246849 podStartE2EDuration="18.666535338s" totalImagesPullingTime="1.937288489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.664500808 +0000 UTC m=+52.776297449" watchObservedRunningTime="2026-09-16 18:44:28.666535338 +0000 UTC m=+52.778331984"
Sep 16 18:44:28.681955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.681915    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xqjz5" podStartSLOduration=16.88006094 podStartE2EDuration="18.681903242s" totalImagesPullingTime="1.801842302s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:28.681055073 +0000 UTC m=+52.792851715" watchObservedRunningTime="2026-09-16 18:44:28.681903242 +0000 UTC m=+52.793699884"
Sep 16 18:44:28.831053 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.831012    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:28.975699 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:28.975620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.074094 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:29.074058    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.174765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:29.174736    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.473895 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:29.473861    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:29.653968 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:29.653939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.256712 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:30.256668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.323200 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:30.323162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.669771 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:30.669731    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.896978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:30.896950    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:30.981524 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:30.981455    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:31.087859 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:31.087809    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:31.201668 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:31.201636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.575862 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:33.575812    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:33.651219 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:33.651189    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:35.567398 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:35.567367    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hqjqz"
Sep 16 18:44:36.126620 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.126588    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-nzwvv"]
Sep 16 18:44:36.129584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.129569    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:36.132555 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.132533    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:44:36.133406 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.133383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mqlmk\""
Sep 16 18:44:36.133674 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.133652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:44:36.153810 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.153784    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-nzwvv"]
Sep 16 18:44:36.159410 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.159389    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5969f9b54c-zrghd"]
Sep 16 18:44:36.193724 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.193691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.233437 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.233391    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"]
Sep 16 18:44:36.235606 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.235585    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59665b5c65-ht8k9"]
Sep 16 18:44:36.235740 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.235721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:36.237563 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.237546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.240112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.240089    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:44:36.240112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.240103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8xfc4\""
Sep 16 18:44:36.241835 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.241804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"]
Sep 16 18:44:36.251668 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.251646    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59665b5c65-ht8k9"]
Sep 16 18:44:36.311887 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.311847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.331716 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.331691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwm2\" (UniqueName: \"kubernetes.io/projected/37a84283-7872-4a59-9f10-fce8fb541c19-kube-api-access-wvwm2\") pod \"downloads-76f99584f4-nzwvv\" (UID: \"37a84283-7872-4a59-9f10-fce8fb541c19\") " pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:36.432002 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.431975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-bound-sa-token\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432125 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-registry-tls\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432125 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/70443ead-2fd9-46cc-b064-e4f63b3cefbf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-44kmx\" (UID: \"70443ead-2fd9-46cc-b064-e4f63b3cefbf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:36.432125 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac21c56-2d25-4921-87c3-16c2485093c1-ca-trust-extracted\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432178    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwm2\" (UniqueName: \"kubernetes.io/projected/37a84283-7872-4a59-9f10-fce8fb541c19-kube-api-access-wvwm2\") pod \"downloads-76f99584f4-nzwvv\" (UID: \"37a84283-7872-4a59-9f10-fce8fb541c19\") " pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:36.432289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac21c56-2d25-4921-87c3-16c2485093c1-registry-certificates\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432237    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcc47\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-kube-api-access-dcc47\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21c56-2d25-4921-87c3-16c2485093c1-trusted-ca\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432488 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432329    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac21c56-2d25-4921-87c3-16c2485093c1-installation-pull-secrets\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.432488 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.432357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac21c56-2d25-4921-87c3-16c2485093c1-image-registry-private-configuration\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.447062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.447040    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 16 18:44:36.457484 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.457467    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 16 18:44:36.468845 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.468827    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwm2\" (UniqueName: \"kubernetes.io/projected/37a84283-7872-4a59-9f10-fce8fb541c19-kube-api-access-wvwm2\") pod \"downloads-76f99584f4-nzwvv\" (UID: \"37a84283-7872-4a59-9f10-fce8fb541c19\") " pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:36.527856 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.527817    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.532776 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.532754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-bound-sa-token\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.532862 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.532785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-registry-tls\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.532862 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.532811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/70443ead-2fd9-46cc-b064-e4f63b3cefbf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-44kmx\" (UID: \"70443ead-2fd9-46cc-b064-e4f63b3cefbf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:36.532961 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.532934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac21c56-2d25-4921-87c3-16c2485093c1-ca-trust-extracted\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533011 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.532986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac21c56-2d25-4921-87c3-16c2485093c1-registry-certificates\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533063 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.533028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcc47\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-kube-api-access-dcc47\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533063 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.533057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21c56-2d25-4921-87c3-16c2485093c1-trusted-ca\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533166 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.533103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac21c56-2d25-4921-87c3-16c2485093c1-installation-pull-secrets\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533166 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.533133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac21c56-2d25-4921-87c3-16c2485093c1-image-registry-private-configuration\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533377 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.533346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3ac21c56-2d25-4921-87c3-16c2485093c1-ca-trust-extracted\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.533768 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.533744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3ac21c56-2d25-4921-87c3-16c2485093c1-registry-certificates\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.534214 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.534191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3ac21c56-2d25-4921-87c3-16c2485093c1-trusted-ca\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.535387 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.535359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-registry-tls\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.535475 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.535366    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3ac21c56-2d25-4921-87c3-16c2485093c1-installation-pull-secrets\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.535592 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.535554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3ac21c56-2d25-4921-87c3-16c2485093c1-image-registry-private-configuration\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.535753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.535735    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:44:36.545738 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.545718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/70443ead-2fd9-46cc-b064-e4f63b3cefbf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-44kmx\" (UID: \"70443ead-2fd9-46cc-b064-e4f63b3cefbf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:36.558966 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.558935    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-bound-sa-token\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.559653 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.559637    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcc47\" (UniqueName: \"kubernetes.io/projected/3ac21c56-2d25-4921-87c3-16c2485093c1-kube-api-access-dcc47\") pod \"image-registry-59665b5c65-ht8k9\" (UID: \"3ac21c56-2d25-4921-87c3-16c2485093c1\") " pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.672185 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.672158    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.745714 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.745649    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mqlmk\""
Sep 16 18:44:36.753610 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.753587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:36.826040 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.826009    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:36.850787 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.850761    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:36.850936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.850914    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-8xfc4\""
Sep 16 18:44:36.856741 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.856504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:36.962290 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:36.962257    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:37.027419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.027343    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-nzwvv"]
Sep 16 18:44:37.029770 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:37.029745    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a84283_7872_4a59_9f10_fce8fb541c19.slice/crio-f56b146d2076450c5d4808e7530578a5cc6ef49d171b62f35d51ee882283f5df WatchSource:0}: Error finding container f56b146d2076450c5d4808e7530578a5cc6ef49d171b62f35d51ee882283f5df: Status 404 returned error can't find the container with id f56b146d2076450c5d4808e7530578a5cc6ef49d171b62f35d51ee882283f5df
Sep 16 18:44:37.122523 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.122476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"]
Sep 16 18:44:37.125327 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:37.125302    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70443ead_2fd9_46cc_b064_e4f63b3cefbf.slice/crio-4e26e44df4165a3b3562fd38c316353609186729d611eca2203f9f1a4acf940f WatchSource:0}: Error finding container 4e26e44df4165a3b3562fd38c316353609186729d611eca2203f9f1a4acf940f: Status 404 returned error can't find the container with id 4e26e44df4165a3b3562fd38c316353609186729d611eca2203f9f1a4acf940f
Sep 16 18:44:37.126360 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.126304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59665b5c65-ht8k9"]
Sep 16 18:44:37.128254 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:37.128234    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac21c56_2d25_4921_87c3_16c2485093c1.slice/crio-5c1b3da2ddff4d942a9bdacb0f316acd0a75e6ed4658b33b92b14ea4cead6f6a WatchSource:0}: Error finding container 5c1b3da2ddff4d942a9bdacb0f316acd0a75e6ed4658b33b92b14ea4cead6f6a: Status 404 returned error can't find the container with id 5c1b3da2ddff4d942a9bdacb0f316acd0a75e6ed4658b33b92b14ea4cead6f6a
Sep 16 18:44:37.679737 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.679698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx" event={"ID":"70443ead-2fd9-46cc-b064-e4f63b3cefbf","Type":"ContainerStarted","Data":"4e26e44df4165a3b3562fd38c316353609186729d611eca2203f9f1a4acf940f"}
Sep 16 18:44:37.681672 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.681636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9" event={"ID":"3ac21c56-2d25-4921-87c3-16c2485093c1","Type":"ContainerStarted","Data":"1ec47aff8658099594a701b3298bec13ad78676781c2bf01841af6368e001018"}
Sep 16 18:44:37.681826 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.681695    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9" event={"ID":"3ac21c56-2d25-4921-87c3-16c2485093c1","Type":"ContainerStarted","Data":"5c1b3da2ddff4d942a9bdacb0f316acd0a75e6ed4658b33b92b14ea4cead6f6a"}
Sep 16 18:44:37.681826 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.681729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:37.682985 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.682962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-nzwvv" event={"ID":"37a84283-7872-4a59-9f10-fce8fb541c19","Type":"ContainerStarted","Data":"f56b146d2076450c5d4808e7530578a5cc6ef49d171b62f35d51ee882283f5df"}
Sep 16 18:44:37.703766 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:37.703626    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9" podStartSLOduration=1.7036102880000001 podStartE2EDuration="1.703610288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:37.702944316 +0000 UTC m=+61.814740956" watchObservedRunningTime="2026-09-16 18:44:37.703610288 +0000 UTC m=+61.815406928"
Sep 16 18:44:38.656370 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:38.656328    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-94j8c"
Sep 16 18:44:38.689125 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:38.689094    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx" event={"ID":"70443ead-2fd9-46cc-b064-e4f63b3cefbf","Type":"ContainerStarted","Data":"b64b0d37b9867a1679ab07ecf8ec2f6f83113bcea4fc3a25ee9892cad2e64283"}
Sep 16 18:44:38.689578 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:38.689446    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:38.694914 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:38.694873    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx"
Sep 16 18:44:38.708004 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:38.707963    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-44kmx" podStartSLOduration=1.320449788 podStartE2EDuration="2.707950111s" totalImagesPullingTime="1.387500323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:38.707160099 +0000 UTC m=+62.818956741" watchObservedRunningTime="2026-09-16 18:44:38.707950111 +0000 UTC m=+62.819746753"
Sep 16 18:44:39.291798 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:39.291754    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:39.376378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:39.376336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:39.561715 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:39.561620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.702389 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:40.702337    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:40.780856 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:40.780817    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.084569 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.084528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:42.087824 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.087803    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:44:42.098215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.098180    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06f9ba4a-9f40-45f7-9ce4-1958a928ff0f-metrics-certs\") pod \"network-metrics-daemon-xxlcv\" (UID: \"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f\") " pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:42.185207 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.184918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:42.185207 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.185135    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.189330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.189309    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:44:42.197786 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.197764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/38006a9b-3fb4-4888-8733-f4468724b1e5-original-pull-secret\") pod \"global-pull-secret-syncer-42hjm\" (UID: \"38006a9b-3fb4-4888-8733-f4468724b1e5\") " pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:42.285398 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.285359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:42.288277 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.288248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:44:42.299241 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.299221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:44:42.309701 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.309676    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j4z\" (UniqueName: \"kubernetes.io/projected/0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd-kube-api-access-m4j4z\") pod \"network-check-target-cv2ng\" (UID: \"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd\") " pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:42.317316 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.317290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-42hjm"
Sep 16 18:44:42.326894 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.326859    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mz62r\""
Sep 16 18:44:42.334746 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.334692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xxlcv"
Sep 16 18:44:42.488683 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.488655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.559474 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.559429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:42.596804 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.596738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-42hjm"]
Sep 16 18:44:42.602264 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:42.602219    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38006a9b_3fb4_4888_8733_f4468724b1e5.slice/crio-5685496b5ceb4cbe9714e93a194513800b062fc7ba23ec0d26b4d3e0cab06739 WatchSource:0}: Error finding container 5685496b5ceb4cbe9714e93a194513800b062fc7ba23ec0d26b4d3e0cab06739: Status 404 returned error can't find the container with id 5685496b5ceb4cbe9714e93a194513800b062fc7ba23ec0d26b4d3e0cab06739
Sep 16 18:44:42.612701 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.612681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fq6l6\""
Sep 16 18:44:42.620010 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.619980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:42.627318 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.627269    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xxlcv"]
Sep 16 18:44:42.629960 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:42.629931    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f9ba4a_9f40_45f7_9ce4_1958a928ff0f.slice/crio-6522b475fadac842c126b21ebcb2785b5b45abade07afc47d0580baab9276c95 WatchSource:0}: Error finding container 6522b475fadac842c126b21ebcb2785b5b45abade07afc47d0580baab9276c95: Status 404 returned error can't find the container with id 6522b475fadac842c126b21ebcb2785b5b45abade07afc47d0580baab9276c95
Sep 16 18:44:42.705142 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.705089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxlcv" event={"ID":"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f","Type":"ContainerStarted","Data":"6522b475fadac842c126b21ebcb2785b5b45abade07afc47d0580baab9276c95"}
Sep 16 18:44:42.706412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.706384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-42hjm" event={"ID":"38006a9b-3fb4-4888-8733-f4468724b1e5","Type":"ContainerStarted","Data":"5685496b5ceb4cbe9714e93a194513800b062fc7ba23ec0d26b4d3e0cab06739"}
Sep 16 18:44:42.899774 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:42.899745    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-cv2ng"]
Sep 16 18:44:42.902795 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:42.902768    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e35e5fb_4dd6_4a58_b717_1cfb3b552cfd.slice/crio-e162a7a7a1d6f1462f2881c154e0434ce9bc15f2e08fa116968a82557df4e8e1 WatchSource:0}: Error finding container e162a7a7a1d6f1462f2881c154e0434ce9bc15f2e08fa116968a82557df4e8e1: Status 404 returned error can't find the container with id e162a7a7a1d6f1462f2881c154e0434ce9bc15f2e08fa116968a82557df4e8e1
Sep 16 18:44:43.715539 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:43.715499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cv2ng" event={"ID":"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd","Type":"ContainerStarted","Data":"e162a7a7a1d6f1462f2881c154e0434ce9bc15f2e08fa116968a82557df4e8e1"}
Sep 16 18:44:44.449591 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.449528    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.538911 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.538786    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.698708 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.698653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.724193 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.724112    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxlcv" event={"ID":"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f","Type":"ContainerStarted","Data":"f503c021fc4b62068e5c37bcaa354e09e5f7005737516158f04039d7f7247942"}
Sep 16 18:44:44.724193 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.724173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xxlcv" event={"ID":"06f9ba4a-9f40-45f7-9ce4-1958a928ff0f","Type":"ContainerStarted","Data":"bdd317b8ee81ef3d1980fbc2c5879a8d5efb659b8dac883093f4be98a01003fd"}
Sep 16 18:44:44.759454 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.759394    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-75kdj"]
Sep 16 18:44:44.762866 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.762408    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xxlcv" podStartSLOduration=66.430426696 podStartE2EDuration="1m7.762386326s" totalImagesPullingTime="1.33195963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:44.75683206 +0000 UTC m=+68.868628698" watchObservedRunningTime="2026-09-16 18:44:44.762386326 +0000 UTC m=+68.874182972"
Sep 16 18:44:44.763610 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.763586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.771510 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.771485    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:44:44.771985 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.771728    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:44:44.772171 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.772005    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rh45j\""
Sep 16 18:44:44.772245 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.772226    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:44:44.772584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.772556    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:44:44.772766 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.772748    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:44:44.773192 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.773167    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.805519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50c34b21-5390-408d-a458-896b801676cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.805572    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bplcc\" (UniqueName: \"kubernetes.io/projected/50c34b21-5390-408d-a458-896b801676cf-kube-api-access-bplcc\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.805605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-tls\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.805631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-node-exporter-wtmp\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.805660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-root\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.805684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50c34b21-5390-408d-a458-896b801676cf-metrics-client-ca\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.806562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-sys\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.806600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50c34b21-5390-408d-a458-896b801676cf-node-exporter-textfile\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.807793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.806641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.833698 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.833543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.907228 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-tls\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-node-exporter-wtmp\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-root\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907295    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50c34b21-5390-408d-a458-896b801676cf-metrics-client-ca\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-sys\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50c34b21-5390-408d-a458-896b801676cf-node-exporter-textfile\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907721 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50c34b21-5390-408d-a458-896b801676cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907721 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.907469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bplcc\" (UniqueName: \"kubernetes.io/projected/50c34b21-5390-408d-a458-896b801676cf-kube-api-access-bplcc\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.907903 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:44.907867    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 16 18:44:44.907969 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:44:44.907947    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-tls podName:50c34b21-5390-408d-a458-896b801676cf nodeName:}" failed. No retries permitted until 2026-09-16 18:44:45.407927344 +0000 UTC m=+69.519723970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-tls") pod "node-exporter-75kdj" (UID: "50c34b21-5390-408d-a458-896b801676cf") : secret "node-exporter-tls" not found
Sep 16 18:44:44.908229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.908210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-node-exporter-wtmp\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.908289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.908264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-root\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.909656 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.909630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50c34b21-5390-408d-a458-896b801676cf-node-exporter-accelerators-collector-config\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.909750 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.909709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50c34b21-5390-408d-a458-896b801676cf-sys\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.909948 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.909929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50c34b21-5390-408d-a458-896b801676cf-node-exporter-textfile\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.910051 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.910035    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50c34b21-5390-408d-a458-896b801676cf-metrics-client-ca\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.931977 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.920395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:44.931977 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.929691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:44.932767 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:44.932710    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bplcc\" (UniqueName: \"kubernetes.io/projected/50c34b21-5390-408d-a458-896b801676cf-kube-api-access-bplcc\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:45.027907 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.027627    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.133357 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.133249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.233582 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.233550    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.338528 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.338402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.413392 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.413357    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-tls\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:45.416229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.416206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50c34b21-5390-408d-a458-896b801676cf-node-exporter-tls\") pod \"node-exporter-75kdj\" (UID: \"50c34b21-5390-408d-a458-896b801676cf\") " pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:45.430365 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.430334    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.677054 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.677021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-75kdj"
Sep 16 18:44:45.779123 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.779091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.877692 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.877650    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:45.907697 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.907661    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:45.910865 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.910837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:45.914511 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.914485    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:44:45.914748 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.914730    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:44:45.915182 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.915162    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:44:45.915298 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.915279    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:44:45.915393 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.915373    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:44:45.915457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.915395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rxw25\""
Sep 16 18:44:45.915974 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.915950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:44:45.916143 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.915953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:44:45.916311 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.916293    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:44:45.916475 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.916453    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:44:45.931654 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.931589    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:45.979790 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:45.979730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:46.017581 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017447    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017581 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017581 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017581 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-web-config\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.017868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp4md\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-kube-api-access-jp4md\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.018172 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017904    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-out\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.018172 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.017936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.018172 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.018002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119451 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-web-config\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.119634 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.120003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119651    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.120003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119665    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.120003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp4md\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-kube-api-access-jp4md\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.120003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-out\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.120003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.119713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.120753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.120410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.121607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.121576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.122681 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.122651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-volume\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.123555 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.123533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.124041 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.123977    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.124041 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.124036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-web-config\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.124158 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.124047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.124701 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.124662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.125222 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.125201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.125387 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.125339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.125499 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.125468    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-out\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.125617 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.125596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.129285 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.129259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp4md\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-kube-api-access-jp4md\") pod \"alertmanager-main-0\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.165912 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.165857    2566 patch_prober.go:28] interesting pod/image-registry-5969f9b54c-zrghd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 16 18:44:46.166078 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.165954    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" podUID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:44:46.223659 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.223590    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:44:46.900944 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:46.900906    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.022131 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:47.022092    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.506474 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:47.506430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.601763 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:47.601724    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.702892 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:47.702855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:47.802072 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:47.801858    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:48.022763 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:48.022722    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:48.665582 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:48.665546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:48.873386 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:48.873344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:48.975152 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:48.975062    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:49.508419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.508304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:49.512442 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.509415    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"]
Sep 16 18:44:49.513979 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.513958    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.516433 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.516409    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:44:49.516549 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.516415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-47p7f\""
Sep 16 18:44:49.519709 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.519683    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"]
Sep 16 18:44:49.550089 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.550058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/76833619-3c79-4c52-aad7-5efe9d093d61-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-z8kgl\" (UID: \"76833619-3c79-4c52-aad7-5efe9d093d61\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.550264 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.550131    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82zwh\" (UniqueName: \"kubernetes.io/projected/76833619-3c79-4c52-aad7-5efe9d093d61-kube-api-access-82zwh\") pod \"monitoring-plugin-5d85778b9d-z8kgl\" (UID: \"76833619-3c79-4c52-aad7-5efe9d093d61\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.572689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.572653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:49.650890 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.650835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82zwh\" (UniqueName: \"kubernetes.io/projected/76833619-3c79-4c52-aad7-5efe9d093d61-kube-api-access-82zwh\") pod \"monitoring-plugin-5d85778b9d-z8kgl\" (UID: \"76833619-3c79-4c52-aad7-5efe9d093d61\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.651084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.650938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/76833619-3c79-4c52-aad7-5efe9d093d61-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-z8kgl\" (UID: \"76833619-3c79-4c52-aad7-5efe9d093d61\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.653746 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.653718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/76833619-3c79-4c52-aad7-5efe9d093d61-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-z8kgl\" (UID: \"76833619-3c79-4c52-aad7-5efe9d093d61\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.659996 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.659969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82zwh\" (UniqueName: \"kubernetes.io/projected/76833619-3c79-4c52-aad7-5efe9d093d61-kube-api-access-82zwh\") pod \"monitoring-plugin-5d85778b9d-z8kgl\" (UID: \"76833619-3c79-4c52-aad7-5efe9d093d61\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:49.825916 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:49.825816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:44:50.972973 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:50.972935    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:51.152706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:51.152663    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:51.253739 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:51.253647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:54.346796 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.346759    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c78b49f6c-4km6l"]
Sep 16 18:44:54.350033 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.350013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.353306 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.353284    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 16 18:44:54.354594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.354272    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 16 18:44:54.354594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.354427    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 16 18:44:54.354594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.354429    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 16 18:44:54.354799 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.354780    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tgftw\""
Sep 16 18:44:54.354799 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.354791    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 16 18:44:54.360297 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.360276    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 16 18:44:54.365659 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.363208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c78b49f6c-4km6l"]
Sep 16 18:44:54.395745 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.395715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-serving-cert\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.395854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.395756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-console-config\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.395854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.395784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-oauth-serving-cert\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.395972 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.395869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-oauth-config\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.395972 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.395929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz2rb\" (UniqueName: \"kubernetes.io/projected/280ec339-bfc8-4675-89da-386baccd0a29-kube-api-access-cz2rb\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.396059 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.395971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-trusted-ca-bundle\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.396059 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.396005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-service-ca\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.496755 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-trusted-ca-bundle\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.496964 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-service-ca\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.496964 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496834    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-serving-cert\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.496964 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-console-config\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.496964 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-oauth-serving-cert\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.496964 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-oauth-config\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.497233 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.496975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz2rb\" (UniqueName: \"kubernetes.io/projected/280ec339-bfc8-4675-89da-386baccd0a29-kube-api-access-cz2rb\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.497685 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.497631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-service-ca\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.497685 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.497659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-console-config\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.497951 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.497922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-trusted-ca-bundle\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.498090 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.497949    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-oauth-serving-cert\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.499777 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.499754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-serving-cert\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.499908 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.499805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-oauth-config\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.508954 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.508930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz2rb\" (UniqueName: \"kubernetes.io/projected/280ec339-bfc8-4675-89da-386baccd0a29-kube-api-access-cz2rb\") pod \"console-6c78b49f6c-4km6l\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") " pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:54.662401 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:54.662366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:44:55.146415 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.146381    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:55.245723 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.245689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:55.453203 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:55.453170    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c34b21_5390_408d_a458_896b801676cf.slice/crio-ac63b3a6a9635f17fef313dea2b0d93dd49889093e28de77016c73dc32e4cfc3 WatchSource:0}: Error finding container ac63b3a6a9635f17fef313dea2b0d93dd49889093e28de77016c73dc32e4cfc3: Status 404 returned error can't find the container with id ac63b3a6a9635f17fef313dea2b0d93dd49889093e28de77016c73dc32e4cfc3
Sep 16 18:44:55.726085 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.726034    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"]
Sep 16 18:44:55.727923 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:55.727891    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76833619_3c79_4c52_aad7_5efe9d093d61.slice/crio-93d503263e67a77c22207abccad717621e58b30e1c8d080453be8300bc88bf45 WatchSource:0}: Error finding container 93d503263e67a77c22207abccad717621e58b30e1c8d080453be8300bc88bf45: Status 404 returned error can't find the container with id 93d503263e67a77c22207abccad717621e58b30e1c8d080453be8300bc88bf45
Sep 16 18:44:55.735776 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.735754    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c78b49f6c-4km6l"]
Sep 16 18:44:55.741086 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:55.739011    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280ec339_bfc8_4675_89da_386baccd0a29.slice/crio-75c7c81021fb195a0d7e99b6fb24495189d9abdd48e8cc50e3fdd1265a54d379 WatchSource:0}: Error finding container 75c7c81021fb195a0d7e99b6fb24495189d9abdd48e8cc50e3fdd1265a54d379: Status 404 returned error can't find the container with id 75c7c81021fb195a0d7e99b6fb24495189d9abdd48e8cc50e3fdd1265a54d379
Sep 16 18:44:55.741086 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.740797    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:44:55.743645 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:55.743621    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a83a4c2_15e8_463e_9e43_dc0669ea3b73.slice/crio-89b3053eb3da859fba9a2f080ed6ea931b446e38bb566a631c4d161ff20767e8 WatchSource:0}: Error finding container 89b3053eb3da859fba9a2f080ed6ea931b446e38bb566a631c4d161ff20767e8: Status 404 returned error can't find the container with id 89b3053eb3da859fba9a2f080ed6ea931b446e38bb566a631c4d161ff20767e8
Sep 16 18:44:55.772956 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.772925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-nzwvv" event={"ID":"37a84283-7872-4a59-9f10-fce8fb541c19","Type":"ContainerStarted","Data":"e9c74a60b5e55cd6b9bc9f991e773398de67bfa2e07ddf9b3d6753226d9d014c"}
Sep 16 18:44:55.773187 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.773171    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:55.774743 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.774724    2566 patch_prober.go:28] interesting pod/downloads-76f99584f4-nzwvv container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused" start-of-body=
Sep 16 18:44:55.774846 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.774776    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-nzwvv" podUID="37a84283-7872-4a59-9f10-fce8fb541c19" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.11:8080/\": dial tcp 10.133.0.11:8080: connect: connection refused"
Sep 16 18:44:55.774966 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.774946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-42hjm" event={"ID":"38006a9b-3fb4-4888-8733-f4468724b1e5","Type":"ContainerStarted","Data":"cad84dc05738a1f519def92be7b128d320be1c6bed0b22f4b1d6aef16a9574f3"}
Sep 16 18:44:55.776238 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.776218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75kdj" event={"ID":"50c34b21-5390-408d-a458-896b801676cf","Type":"ContainerStarted","Data":"ac63b3a6a9635f17fef313dea2b0d93dd49889093e28de77016c73dc32e4cfc3"}
Sep 16 18:44:55.777540 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.777520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c78b49f6c-4km6l" event={"ID":"280ec339-bfc8-4675-89da-386baccd0a29","Type":"ContainerStarted","Data":"75c7c81021fb195a0d7e99b6fb24495189d9abdd48e8cc50e3fdd1265a54d379"}
Sep 16 18:44:55.778696 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.778664    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl" event={"ID":"76833619-3c79-4c52-aad7-5efe9d093d61","Type":"ContainerStarted","Data":"93d503263e67a77c22207abccad717621e58b30e1c8d080453be8300bc88bf45"}
Sep 16 18:44:55.779922 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.779900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"89b3053eb3da859fba9a2f080ed6ea931b446e38bb566a631c4d161ff20767e8"}
Sep 16 18:44:55.781361 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.781338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-cv2ng" event={"ID":"0e35e5fb-4dd6-4a58-b717-1cfb3b552cfd","Type":"ContainerStarted","Data":"53448aa75cca382c6bf4472c3630a208b76a77b996723554b685f72992dd4b9d"}
Sep 16 18:44:55.781471 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.781458    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:44:55.817111 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.817026    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-nzwvv" podStartSLOduration=1.217274697 podStartE2EDuration="19.817005073s" totalImagesPullingTime="18.599730376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.81555647 +0000 UTC m=+79.927353114" watchObservedRunningTime="2026-09-16 18:44:55.817005073 +0000 UTC m=+79.928801716"
Sep 16 18:44:55.874388 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.872868    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-42hjm" podStartSLOduration=65.89738964 podStartE2EDuration="1m18.872847221s" totalImagesPullingTime="12.975457581s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.854206122 +0000 UTC m=+79.966002763" watchObservedRunningTime="2026-09-16 18:44:55.872847221 +0000 UTC m=+79.984643863"
Sep 16 18:44:55.874388 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.873152    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-cv2ng" podStartSLOduration=66.325770498 podStartE2EDuration="1m18.873141681s" totalImagesPullingTime="12.547371183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:43:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:55.872325917 +0000 UTC m=+79.984122559" watchObservedRunningTime="2026-09-16 18:44:55.873141681 +0000 UTC m=+79.984938326"
Sep 16 18:44:55.938943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.938699    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:55.981942 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:55.981911    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c78b49f6c-4km6l"]
Sep 16 18:44:56.014857 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.014766    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.014857 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.014837    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f8884b7-bzsz2"]
Sep 16 18:44:56.018079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.018057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.030162 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.030123    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8884b7-bzsz2"]
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.110924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-trusted-ca-bundle\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.110973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-config\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.111005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-oauth-serving-cert\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.111043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fzrq\" (UniqueName: \"kubernetes.io/projected/d4624f55-bf0f-4b9e-9118-2caabf792a1e-kube-api-access-6fzrq\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.111144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-serving-cert\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.111194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-oauth-config\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.111346 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.111220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-service-ca\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.167690 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.167433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212547    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-oauth-config\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-service-ca\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-trusted-ca-bundle\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-config\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-oauth-serving-cert\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fzrq\" (UniqueName: \"kubernetes.io/projected/d4624f55-bf0f-4b9e-9118-2caabf792a1e-kube-api-access-6fzrq\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.213595 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.212833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-serving-cert\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.214624 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.214289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-trusted-ca-bundle\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.218913 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.215342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-oauth-serving-cert\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.218913 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.217456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-config\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.218913 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.218355    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-service-ca\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.222931 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.219068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-oauth-config\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.222931 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.221691    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-serving-cert\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.232760 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.232732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fzrq\" (UniqueName: \"kubernetes.io/projected/d4624f55-bf0f-4b9e-9118-2caabf792a1e-kube-api-access-6fzrq\") pod \"console-5f8884b7-bzsz2\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") " pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.331255 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.331215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:44:56.585799 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.585633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.685781 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.685720    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:56.792259 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:56.792053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-nzwvv"
Sep 16 18:44:57.046830 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:57.046766    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f8884b7-bzsz2"]
Sep 16 18:44:57.464813 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:44:57.464747    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4624f55_bf0f_4b9e_9118_2caabf792a1e.slice/crio-fb88f2009113114bd661ec95f6eac6cd366f59eff3e5141420a58d96e0d898f2 WatchSource:0}: Error finding container fb88f2009113114bd661ec95f6eac6cd366f59eff3e5141420a58d96e0d898f2: Status 404 returned error can't find the container with id fb88f2009113114bd661ec95f6eac6cd366f59eff3e5141420a58d96e0d898f2
Sep 16 18:44:57.797062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:57.796951    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8884b7-bzsz2" event={"ID":"d4624f55-bf0f-4b9e-9118-2caabf792a1e","Type":"ContainerStarted","Data":"fb88f2009113114bd661ec95f6eac6cd366f59eff3e5141420a58d96e0d898f2"}
Sep 16 18:44:58.703326 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:58.703292    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59665b5c65-ht8k9"
Sep 16 18:44:58.944703 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:58.944639    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:59.056552 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:59.056425    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:59.314703 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:59.314548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:44:59.418728 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:44:59.418651    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.265948 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.265907    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.349271 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.349211    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.813180 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.813149    2566 generic.go:356] "Generic (PLEG): container finished" podID="50c34b21-5390-408d-a458-896b801676cf" containerID="1a47090a816e12b13e40976c34ba721a0d2cce5c03258108e1b66c7cc9395112" exitCode=0
Sep 16 18:45:00.813290 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.813259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75kdj" event={"ID":"50c34b21-5390-408d-a458-896b801676cf","Type":"ContainerDied","Data":"1a47090a816e12b13e40976c34ba721a0d2cce5c03258108e1b66c7cc9395112"}
Sep 16 18:45:00.816185 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.816025    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:00.816544 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.816491    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c78b49f6c-4km6l" event={"ID":"280ec339-bfc8-4675-89da-386baccd0a29","Type":"ContainerStarted","Data":"fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236"}
Sep 16 18:45:00.818999 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.818972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8884b7-bzsz2" event={"ID":"d4624f55-bf0f-4b9e-9118-2caabf792a1e","Type":"ContainerStarted","Data":"791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a"}
Sep 16 18:45:00.821124 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.821099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl" event={"ID":"76833619-3c79-4c52-aad7-5efe9d093d61","Type":"ContainerStarted","Data":"e9e07c09ac92f66ce2479187f3ba2eb6eaf2f7b24395abd7af3102f44c00ceb4"}
Sep 16 18:45:00.821562 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.821508    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:45:00.824663 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.824195    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893" exitCode=0
Sep 16 18:45:00.824663 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.824244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"}
Sep 16 18:45:00.828986 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.828938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl"
Sep 16 18:45:00.894608 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.894359    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-z8kgl" podStartSLOduration=7.39479047 podStartE2EDuration="11.894339727s" totalImagesPullingTime="4.499549257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:00.860212004 +0000 UTC m=+84.972008646" watchObservedRunningTime="2026-09-16 18:45:00.894339727 +0000 UTC m=+85.006136370"
Sep 16 18:45:00.894608 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.894462    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c78b49f6c-4km6l" podStartSLOduration=2.053068981 podStartE2EDuration="6.894455001s" totalImagesPullingTime="4.84138602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:00.892009537 +0000 UTC m=+85.003806219" watchObservedRunningTime="2026-09-16 18:45:00.894455001 +0000 UTC m=+85.006251641"
Sep 16 18:45:00.931901 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:00.926192    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:01.178190 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.178128    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" podUID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" containerName="registry" containerID="cri-o://c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867" gracePeriod=30
Sep 16 18:45:01.466813 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.466786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:45:01.563743 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.563709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-trusted-ca\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.563965 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.563805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-certificates\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.563965 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.563850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-installation-pull-secrets\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.563965 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.563903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.563965 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.563957    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-bound-sa-token\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.564190 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.564002    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-ca-trust-extracted\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.564190 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.564038    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdp9x\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-kube-api-access-wdp9x\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.564190 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.564071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-image-registry-private-configuration\") pod \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\" (UID: \"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b\") "
Sep 16 18:45:01.564343 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.564198    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:01.564343 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.564314    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-trusted-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.564519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.564473    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:01.567905 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.567845    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-kube-api-access-wdp9x" (OuterVolumeSpecName: "kube-api-access-wdp9x") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "kube-api-access-wdp9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:01.568156 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.568120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:01.568330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.568295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:01.568330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.568309    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:01.568456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.568413    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:01.575385 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.575327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" (UID: "a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:45:01.665722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665685    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-certificates\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.665722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665722    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-installation-pull-secrets\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.666045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665736    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-registry-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.666045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665750    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-bound-sa-token\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.666045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665761    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-ca-trust-extracted\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.666045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665773    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdp9x\" (UniqueName: \"kubernetes.io/projected/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-kube-api-access-wdp9x\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.666045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.665789    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b-image-registry-private-configuration\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:01.833260 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.832286    2566 generic.go:356] "Generic (PLEG): container finished" podID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" containerID="c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867" exitCode=0
Sep 16 18:45:01.833260 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.832468    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd"
Sep 16 18:45:01.834727 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.834693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" event={"ID":"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b","Type":"ContainerDied","Data":"c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867"}
Sep 16 18:45:01.834933 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.834753    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5969f9b54c-zrghd" event={"ID":"a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b","Type":"ContainerDied","Data":"4e953c1539d0d8f14999ff9a4cf44587be9059c1426fe78649e77ff1575fbc2f"}
Sep 16 18:45:01.834933 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.834782    2566 scope.go:117] "RemoveContainer" containerID="c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867"
Sep 16 18:45:01.839579 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.838693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75kdj" event={"ID":"50c34b21-5390-408d-a458-896b801676cf","Type":"ContainerStarted","Data":"2d20a778cb118f873bd3937f4d410d7b8612cb77de337a46795e1edd7e6aac99"}
Sep 16 18:45:01.839579 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.838766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-75kdj" event={"ID":"50c34b21-5390-408d-a458-896b801676cf","Type":"ContainerStarted","Data":"9a9111b77327f00b783caf8179ef3201cbd2cf840a52bfec58529ff76c1dd1e4"}
Sep 16 18:45:01.852542 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.852520    2566 scope.go:117] "RemoveContainer" containerID="c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867"
Sep 16 18:45:01.852869 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:45:01.852841    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867\": container with ID starting with c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867 not found: ID does not exist" containerID="c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867"
Sep 16 18:45:01.853000 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.852871    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867"} err="failed to get container status \"c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867\": rpc error: code = NotFound desc = could not find container \"c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867\": container with ID starting with c3e0b20acef89071f1d335d33e977c88ae9b7aef9e92058dffa655a666e52867 not found: ID does not exist"
Sep 16 18:45:01.869950 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.869904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-75kdj" podStartSLOduration=13.009707028 podStartE2EDuration="17.869867386s" totalImagesPullingTime="4.772700594s" totalInitContainerRuntime="87.459764ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.868324132 +0000 UTC m=+85.980120776" watchObservedRunningTime="2026-09-16 18:45:01.869867386 +0000 UTC m=+85.981664026"
Sep 16 18:45:01.898440 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.898392    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f8884b7-bzsz2" podStartSLOduration=3.671335178 podStartE2EDuration="6.898372432s" totalImagesPullingTime="3.227037254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:01.897338343 +0000 UTC m=+86.009134989" watchObservedRunningTime="2026-09-16 18:45:01.898372432 +0000 UTC m=+86.010169074"
Sep 16 18:45:01.919329 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.919304    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5969f9b54c-zrghd"]
Sep 16 18:45:01.932609 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:01.932581    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5969f9b54c-zrghd"]
Sep 16 18:45:02.402831 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:02.402789    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" path="/var/lib/kubelet/pods/a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b/volumes"
Sep 16 18:45:03.571123 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.571094    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:03.645776 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.645676    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:03.853149 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.853074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"}
Sep 16 18:45:03.853325 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.853193    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"}
Sep 16 18:45:03.853325 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.853239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"}
Sep 16 18:45:03.853325 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.853274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"}
Sep 16 18:45:03.853325 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:03.853306    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"}
Sep 16 18:45:04.663303 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:04.663265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:45:05.864661 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:05.864616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerStarted","Data":"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"}
Sep 16 18:45:05.896797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:05.896744    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=12.978219443 podStartE2EDuration="20.896725807s" totalImagesPullingTime="7.895361017s" totalInitContainerRuntime="23.145347ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:05.894626687 +0000 UTC m=+90.006423328" watchObservedRunningTime="2026-09-16 18:45:05.896725807 +0000 UTC m=+90.008522448"
Sep 16 18:45:06.331935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:06.331867    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:45:06.332143 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:06.331969    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:45:06.336626 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:06.336598    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:45:06.871573 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:06.871540    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:45:07.059467 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:07.059434    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.164856 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:07.164818    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:07.293573 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:07.293536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:10.936972 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:10.936938    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.037150 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:11.037099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:11.106586 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:11.106540    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:18.589201 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:18.589166    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:18.691853 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:18.691810    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.350462 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:21.350427    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:21.452273 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:21.452238    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:25.843847 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:25.843786    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c78b49f6c-4km6l" podUID="280ec339-bfc8-4675-89da-386baccd0a29" containerName="console" containerID="cri-o://fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236" gracePeriod=15
Sep 16 18:45:26.098380 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.098312    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:45:26.164109 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164068    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-console-config\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.164109 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-oauth-config\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.164330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-trusted-ca-bundle\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.164330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164194    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-service-ca\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.164330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-oauth-serving-cert\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.164577 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164530    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-console-config" (OuterVolumeSpecName: "console-config") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:26.164724 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164673    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:26.164724 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164678    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:26.164724 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.164695    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-service-ca" (OuterVolumeSpecName: "service-ca") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:45:26.166486 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.166464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:26.265298 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265256    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-serving-cert\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.265477 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265440    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cz2rb\" (UniqueName: \"kubernetes.io/projected/280ec339-bfc8-4675-89da-386baccd0a29-kube-api-access-cz2rb\") pod \"280ec339-bfc8-4675-89da-386baccd0a29\" (UID: \"280ec339-bfc8-4675-89da-386baccd0a29\") "
Sep 16 18:45:26.265648 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265628    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.265713 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265653    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.265713 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265666    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.265713 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265677    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/280ec339-bfc8-4675-89da-386baccd0a29-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.265713 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.265685    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.267396 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.267371    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:45:26.267510 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.267483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280ec339-bfc8-4675-89da-386baccd0a29-kube-api-access-cz2rb" (OuterVolumeSpecName: "kube-api-access-cz2rb") pod "280ec339-bfc8-4675-89da-386baccd0a29" (UID: "280ec339-bfc8-4675-89da-386baccd0a29"). InnerVolumeSpecName "kube-api-access-cz2rb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:45:26.366936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.366857    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz2rb\" (UniqueName: \"kubernetes.io/projected/280ec339-bfc8-4675-89da-386baccd0a29-kube-api-access-cz2rb\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.366936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.366903    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/280ec339-bfc8-4675-89da-386baccd0a29-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:45:26.791979 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.791950    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-cv2ng"
Sep 16 18:45:26.946037 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.946006    2566 generic.go:356] "Generic (PLEG): container finished" podID="280ec339-bfc8-4675-89da-386baccd0a29" containerID="fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236" exitCode=0
Sep 16 18:45:26.946435 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.946053    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c78b49f6c-4km6l"
Sep 16 18:45:26.946435 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.946084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c78b49f6c-4km6l" event={"ID":"280ec339-bfc8-4675-89da-386baccd0a29","Type":"ContainerDied","Data":"fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236"}
Sep 16 18:45:26.946435 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.946136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c78b49f6c-4km6l" event={"ID":"280ec339-bfc8-4675-89da-386baccd0a29","Type":"ContainerDied","Data":"75c7c81021fb195a0d7e99b6fb24495189d9abdd48e8cc50e3fdd1265a54d379"}
Sep 16 18:45:26.946435 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.946166    2566 scope.go:117] "RemoveContainer" containerID="fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236"
Sep 16 18:45:26.962591 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.962566    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c78b49f6c-4km6l"]
Sep 16 18:45:26.963979 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.963958    2566 scope.go:117] "RemoveContainer" containerID="fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236"
Sep 16 18:45:26.964392 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:45:26.964371    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236\": container with ID starting with fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236 not found: ID does not exist" containerID="fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236"
Sep 16 18:45:26.964470 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.964401    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236"} err="failed to get container status \"fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236\": rpc error: code = NotFound desc = could not find container \"fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236\": container with ID starting with fd7874489515ed7ab78c1058be65fff49e1ea9f6468c47e348e18f57dbea2236 not found: ID does not exist"
Sep 16 18:45:26.968163 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:26.968143    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c78b49f6c-4km6l"]
Sep 16 18:45:28.403348 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:28.403298    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="280ec339-bfc8-4675-89da-386baccd0a29" path="/var/lib/kubelet/pods/280ec339-bfc8-4675-89da-386baccd0a29/volumes"
Sep 16 18:45:28.659753 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:28.659656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:45:28.724709 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:45:28.724671    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.871399 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:00.871365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:00.945376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:00.945345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:05.151651 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.151617    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:05.152062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.151955    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-metric" containerID="cri-o://5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab" gracePeriod=120
Sep 16 18:46:05.152062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.151995    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="alertmanager" containerID="cri-o://4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba" gracePeriod=120
Sep 16 18:46:05.152062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.151965    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="prom-label-proxy" containerID="cri-o://d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0" gracePeriod=120
Sep 16 18:46:05.152062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.151955    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="config-reloader" containerID="cri-o://99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a" gracePeriod=120
Sep 16 18:46:05.152318 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.152079    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-web" containerID="cri-o://5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1" gracePeriod=120
Sep 16 18:46:05.152318 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:05.152154    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy" containerID="cri-o://d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9" gracePeriod=120
Sep 16 18:46:06.090296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090262    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0" exitCode=0
Sep 16 18:46:06.090296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090290    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9" exitCode=0
Sep 16 18:46:06.090296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090299    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a" exitCode=0
Sep 16 18:46:06.090296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090306    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba" exitCode=0
Sep 16 18:46:06.090583 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"}
Sep 16 18:46:06.090583 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"}
Sep 16 18:46:06.090583 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"}
Sep 16 18:46:06.090583 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.090458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"}
Sep 16 18:46:06.392156 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.392133    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:06.454426 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454400    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454438    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-trusted-ca-bundle\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454468    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-main-tls\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454493    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-volume\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-cluster-tls-config\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454617    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454660    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-metrics-client-ca\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-web-config\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jp4md\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-kube-api-access-jp4md\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.454797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-tls-assets\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.455066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-out\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.455066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-web\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.455066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454915    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-main-db\") pod \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\" (UID: \"4a83a4c2-15e8-463e-9e43-dc0669ea3b73\") "
Sep 16 18:46:06.455066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.454948    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:06.455066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.455045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:06.455310 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.455201    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.455310 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.455221    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-metrics-client-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.456356 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.456310    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:06.458666 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.458610    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-volume" (OuterVolumeSpecName: "config-volume") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.458795 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.458770    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.458862 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.458775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-out" (OuterVolumeSpecName: "config-out") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:06.458990 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.458952    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.459174 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.459141    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.459287 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.459217    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:06.459515 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.459488    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.459667 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.459642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-kube-api-access-jp4md" (OuterVolumeSpecName: "kube-api-access-jp4md") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "kube-api-access-jp4md". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:06.462377 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.462234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.468953 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.468928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-web-config" (OuterVolumeSpecName: "web-config") pod "4a83a4c2-15e8-463e-9e43-dc0669ea3b73" (UID: "4a83a4c2-15e8-463e-9e43-dc0669ea3b73"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:06.556472 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556435    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556472 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556465    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-main-tls\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556472 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556475    2566 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-volume\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556485    2566 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-cluster-tls-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556494    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556503    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-web-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556512    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp4md\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-kube-api-access-jp4md\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556520    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-tls-assets\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556529    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-config-out\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556537    2566 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:06.556689 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:06.556547    2566 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4a83a4c2-15e8-463e-9e43-dc0669ea3b73-alertmanager-main-db\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:07.097485 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097451    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab" exitCode=0
Sep 16 18:46:07.097485 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097476    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerID="5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1" exitCode=0
Sep 16 18:46:07.097680 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"}
Sep 16 18:46:07.097680 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097549    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.097680 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"}
Sep 16 18:46:07.097680 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4a83a4c2-15e8-463e-9e43-dc0669ea3b73","Type":"ContainerDied","Data":"89b3053eb3da859fba9a2f080ed6ea931b446e38bb566a631c4d161ff20767e8"}
Sep 16 18:46:07.097810 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.097697    2566 scope.go:117] "RemoveContainer" containerID="d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"
Sep 16 18:46:07.105273 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.105210    2566 scope.go:117] "RemoveContainer" containerID="5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"
Sep 16 18:46:07.111720 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.111701    2566 scope.go:117] "RemoveContainer" containerID="d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"
Sep 16 18:46:07.118022 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.118006    2566 scope.go:117] "RemoveContainer" containerID="5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"
Sep 16 18:46:07.122124 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.122098    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:07.125156 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.125138    2566 scope.go:117] "RemoveContainer" containerID="99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"
Sep 16 18:46:07.126489 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.126470    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:07.131610 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.131595    2566 scope.go:117] "RemoveContainer" containerID="4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"
Sep 16 18:46:07.138300 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.138280    2566 scope.go:117] "RemoveContainer" containerID="9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"
Sep 16 18:46:07.144890 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.144862    2566 scope.go:117] "RemoveContainer" containerID="d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"
Sep 16 18:46:07.145140 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.145122    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0\": container with ID starting with d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0 not found: ID does not exist" containerID="d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"
Sep 16 18:46:07.145192 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145149    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"} err="failed to get container status \"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0\": rpc error: code = NotFound desc = could not find container \"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0\": container with ID starting with d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0 not found: ID does not exist"
Sep 16 18:46:07.145192 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145165    2566 scope.go:117] "RemoveContainer" containerID="5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"
Sep 16 18:46:07.145392 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.145375    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab\": container with ID starting with 5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab not found: ID does not exist" containerID="5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"
Sep 16 18:46:07.145438 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145397    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"} err="failed to get container status \"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab\": rpc error: code = NotFound desc = could not find container \"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab\": container with ID starting with 5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab not found: ID does not exist"
Sep 16 18:46:07.145438 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145420    2566 scope.go:117] "RemoveContainer" containerID="d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"
Sep 16 18:46:07.145636 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.145619    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9\": container with ID starting with d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9 not found: ID does not exist" containerID="d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"
Sep 16 18:46:07.145681 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145639    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"} err="failed to get container status \"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9\": rpc error: code = NotFound desc = could not find container \"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9\": container with ID starting with d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9 not found: ID does not exist"
Sep 16 18:46:07.145681 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145650    2566 scope.go:117] "RemoveContainer" containerID="5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"
Sep 16 18:46:07.145899 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.145867    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1\": container with ID starting with 5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1 not found: ID does not exist" containerID="5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"
Sep 16 18:46:07.145943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145904    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"} err="failed to get container status \"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1\": rpc error: code = NotFound desc = could not find container \"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1\": container with ID starting with 5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1 not found: ID does not exist"
Sep 16 18:46:07.145943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.145919    2566 scope.go:117] "RemoveContainer" containerID="99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"
Sep 16 18:46:07.146151 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.146130    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a\": container with ID starting with 99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a not found: ID does not exist" containerID="99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"
Sep 16 18:46:07.146191 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.146157    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"} err="failed to get container status \"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a\": rpc error: code = NotFound desc = could not find container \"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a\": container with ID starting with 99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a not found: ID does not exist"
Sep 16 18:46:07.146191 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.146169    2566 scope.go:117] "RemoveContainer" containerID="4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.146534    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba\": container with ID starting with 4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba not found: ID does not exist" containerID="4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.146562    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"} err="failed to get container status \"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba\": rpc error: code = NotFound desc = could not find container \"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba\": container with ID starting with 4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.146586    2566 scope.go:117] "RemoveContainer" containerID="9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:07.146962    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893\": container with ID starting with 9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893 not found: ID does not exist" containerID="9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.146984    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"} err="failed to get container status \"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893\": rpc error: code = NotFound desc = could not find container \"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893\": container with ID starting with 9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893 not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147001    2566 scope.go:117] "RemoveContainer" containerID="d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147273    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0"} err="failed to get container status \"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0\": rpc error: code = NotFound desc = could not find container \"d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0\": container with ID starting with d7affc4d9539aa6556feafa964b6dbdfab401f44d1f60e91d7c33f4ffafa21f0 not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147299    2566 scope.go:117] "RemoveContainer" containerID="5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147632    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab"} err="failed to get container status \"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab\": rpc error: code = NotFound desc = could not find container \"5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab\": container with ID starting with 5c767b0a1c1406a4926bc2d891d02cdb99157a47aa4b1e8ca87622e73ad341ab not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147650    2566 scope.go:117] "RemoveContainer" containerID="d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147947    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9"} err="failed to get container status \"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9\": rpc error: code = NotFound desc = could not find container \"d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9\": container with ID starting with d6746a590a6ad27ca265ae66955f9d497287755d66981b484df2c7f0fc3fcfd9 not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.147972    2566 scope.go:117] "RemoveContainer" containerID="5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.148234    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1"} err="failed to get container status \"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1\": rpc error: code = NotFound desc = could not find container \"5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1\": container with ID starting with 5dcb1e2e2a8f31eec2f3c801ee980dd7ed04a8d36207c25b0419500bf61075f1 not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.148260    2566 scope.go:117] "RemoveContainer" containerID="99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.148516    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a"} err="failed to get container status \"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a\": rpc error: code = NotFound desc = could not find container \"99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a\": container with ID starting with 99e173b27f4b1f3533fe5159102678a5de8ea75f1760497a9cd1d546f780755a not found: ID does not exist"
Sep 16 18:46:07.148627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.148539    2566 scope.go:117] "RemoveContainer" containerID="4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"
Sep 16 18:46:07.149247 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.148806    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba"} err="failed to get container status \"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba\": rpc error: code = NotFound desc = could not find container \"4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba\": container with ID starting with 4713c6ab5d59cf1fcec46d5dabc5b0cca89973c84329e9847058d83c057075ba not found: ID does not exist"
Sep 16 18:46:07.149247 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.148835    2566 scope.go:117] "RemoveContainer" containerID="9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"
Sep 16 18:46:07.149318 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.149249    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893"} err="failed to get container status \"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893\": rpc error: code = NotFound desc = could not find container \"9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893\": container with ID starting with 9505be5c46469a55aa2c689ebbae83404a7fa36ae2a38965b1f34bc0f9982893 not found: ID does not exist"
Sep 16 18:46:07.151502 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151481    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:07.151708 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151695    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151709    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151720    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="init-config-reloader"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151725    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="init-config-reloader"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151732    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="alertmanager"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151737    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="alertmanager"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151742    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-metric"
Sep 16 18:46:07.151747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151749    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-metric"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151758    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="config-reloader"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151763    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="config-reloader"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151769    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-web"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151774    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-web"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151779    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="prom-label-proxy"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151784    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="prom-label-proxy"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151793    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" containerName="registry"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151797    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" containerName="registry"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280ec339-bfc8-4675-89da-386baccd0a29" containerName="console"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151809    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="280ec339-bfc8-4675-89da-386baccd0a29" containerName="console"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151861    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ed46c3-8dce-4fe8-a9cf-0406d4440b3b" containerName="registry"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151868    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="config-reloader"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151873    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-web"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151894    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy-metric"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151900    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="280ec339-bfc8-4675-89da-386baccd0a29" containerName="console"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151906    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="kube-rbac-proxy"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151910    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="alertmanager"
Sep 16 18:46:07.151955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.151916    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" containerName="prom-label-proxy"
Sep 16 18:46:07.154922 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.154906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.157649 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.157632    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:46:07.157755 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.157631    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:46:07.157927 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.157911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:46:07.158007 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.157955    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:46:07.158007 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.157982    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rxw25\""
Sep 16 18:46:07.158177 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.158163    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:46:07.158244 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.158169    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:46:07.158324 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.158305    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:46:07.158467 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.158454    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:46:07.162591 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.162573    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:46:07.172022 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.171998    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:07.261411 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261603 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261516    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fb1523-841c-45aa-a101-2c2612d9b1d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261603 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261545    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fb1523-841c-45aa-a101-2c2612d9b1d6-config-out\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261603 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261603 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fb1523-841c-45aa-a101-2c2612d9b1d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261618    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-web-config\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzlcl\" (UniqueName: \"kubernetes.io/projected/b4fb1523-841c-45aa-a101-2c2612d9b1d6-kube-api-access-mzlcl\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fb1523-841c-45aa-a101-2c2612d9b1d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.261722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.261710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb1523-841c-45aa-a101-2c2612d9b1d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362668 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362668 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-web-config\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362668 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzlcl\" (UniqueName: \"kubernetes.io/projected/b4fb1523-841c-45aa-a101-2c2612d9b1d6-kube-api-access-mzlcl\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fb1523-841c-45aa-a101-2c2612d9b1d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb1523-841c-45aa-a101-2c2612d9b1d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362759    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.362864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362843    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fb1523-841c-45aa-a101-2c2612d9b1d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.363229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fb1523-841c-45aa-a101-2c2612d9b1d6-config-out\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.363229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.363229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.362954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fb1523-841c-45aa-a101-2c2612d9b1d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.363229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.363092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b4fb1523-841c-45aa-a101-2c2612d9b1d6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.363600 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.363574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb1523-841c-45aa-a101-2c2612d9b1d6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.364219 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.364196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4fb1523-841c-45aa-a101-2c2612d9b1d6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.365760 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.365643    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-web-config\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.365760 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.365718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4fb1523-841c-45aa-a101-2c2612d9b1d6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.365935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.365797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.365935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.365822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4fb1523-841c-45aa-a101-2c2612d9b1d6-config-out\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.365935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.365830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.366057 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.365969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.366338 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.366317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-config-volume\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.367153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.367132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.367598 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.367572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b4fb1523-841c-45aa-a101-2c2612d9b1d6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.371198 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.371181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzlcl\" (UniqueName: \"kubernetes.io/projected/b4fb1523-841c-45aa-a101-2c2612d9b1d6-kube-api-access-mzlcl\") pod \"alertmanager-main-0\" (UID: \"b4fb1523-841c-45aa-a101-2c2612d9b1d6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.463520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.463489    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:07.741931 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:07.741810    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:07.746668 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:46:07.746643    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4fb1523_841c_45aa_a101_2c2612d9b1d6.slice/crio-097dc4c71f4f0ce960588997612a0aabdcb2587552eb600840938a1e9c27d8a3 WatchSource:0}: Error finding container 097dc4c71f4f0ce960588997612a0aabdcb2587552eb600840938a1e9c27d8a3: Status 404 returned error can't find the container with id 097dc4c71f4f0ce960588997612a0aabdcb2587552eb600840938a1e9c27d8a3
Sep 16 18:46:08.102765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:08.102693    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4fb1523-841c-45aa-a101-2c2612d9b1d6" containerID="03b08fc92b8564b6204373f567d2612025e2abbd05f36d20291596945e3bac9b" exitCode=0
Sep 16 18:46:08.102934 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:08.102782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerDied","Data":"03b08fc92b8564b6204373f567d2612025e2abbd05f36d20291596945e3bac9b"}
Sep 16 18:46:08.102934 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:08.102870    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"097dc4c71f4f0ce960588997612a0aabdcb2587552eb600840938a1e9c27d8a3"}
Sep 16 18:46:08.404814 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:08.404788    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a83a4c2-15e8-463e-9e43-dc0669ea3b73" path="/var/lib/kubelet/pods/4a83a4c2-15e8-463e-9e43-dc0669ea3b73/volumes"
Sep 16 18:46:09.110665 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.110632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"0bb4d88391c0b1f59cd513224fade2f99114c27abcde996af4359360ad1c4485"}
Sep 16 18:46:09.111065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.110696    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"2f17f1398955758870fe59b4b08e6cd36f68e8b29e8c36d2a3a32500d706e24b"}
Sep 16 18:46:09.111065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.110722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"bb72f43ccceaf46b84f61ecc2c3d6254be5cae89dd88a810597aa858240cb0ac"}
Sep 16 18:46:09.111065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.110749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"3e5e8b0da00fcc018fa36777da05f44b83c216f2cfb680dd3a47670f17d17eb3"}
Sep 16 18:46:09.111065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.110777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"ce41bf8e5911b3927e95a5b44ad52a1ffe15978130cdde130dce2a73afe38a02"}
Sep 16 18:46:09.111065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.110800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b4fb1523-841c-45aa-a101-2c2612d9b1d6","Type":"ContainerStarted","Data":"e4f859a0dbc19914da55dc49fe0e2cbce3993219b87d694bd599cbedf9ba81f0"}
Sep 16 18:46:09.139275 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:09.139209    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.122076694 podStartE2EDuration="2.139190606s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.113912ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:09.138601738 +0000 UTC m=+153.250398390" watchObservedRunningTime="2026-09-16 18:46:09.139190606 +0000 UTC m=+153.250987248"
Sep 16 18:46:12.769509 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:12.769473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.889045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:12.889015    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-648469b4d6-mb9jb"]
Sep 16 18:46:12.891368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:12.891348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:12.894501 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:12.894477    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:12.915379 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:12.915353    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648469b4d6-mb9jb"]
Sep 16 18:46:12.993330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:12.993298    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:13.007236 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-trusted-ca-bundle\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.007300 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-oauth-serving-cert\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.007300 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-serving-cert\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.007367 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-service-ca\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.007367 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-oauth-config\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.007423 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-console-config\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.007457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.007438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttw9x\" (UniqueName: \"kubernetes.io/projected/e67eb7c8-649e-4d76-a7af-9176b7997839-kube-api-access-ttw9x\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108513 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-service-ca\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108513 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-oauth-config\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108513 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-console-config\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108667 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttw9x\" (UniqueName: \"kubernetes.io/projected/e67eb7c8-649e-4d76-a7af-9176b7997839-kube-api-access-ttw9x\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108667 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-trusted-ca-bundle\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108749 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-oauth-serving-cert\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.108749 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.108731    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-serving-cert\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.109248 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.109226    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-console-config\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.109248 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.109234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-service-ca\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.109428 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.109395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-oauth-serving-cert\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.109706 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.109679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-trusted-ca-bundle\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.111073 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.111047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-oauth-config\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.111277 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.111255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-serving-cert\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.118453 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.118433    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttw9x\" (UniqueName: \"kubernetes.io/projected/e67eb7c8-649e-4d76-a7af-9176b7997839-kube-api-access-ttw9x\") pod \"console-648469b4d6-mb9jb\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") " pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.200233 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.200198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:13.471011 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:13.470987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-648469b4d6-mb9jb"]
Sep 16 18:46:13.473266 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:46:13.473243    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode67eb7c8_649e_4d76_a7af_9176b7997839.slice/crio-d1f866b1693c62835c88b95333e77ef651de3437918975608e3175be38e02820 WatchSource:0}: Error finding container d1f866b1693c62835c88b95333e77ef651de3437918975608e3175be38e02820: Status 404 returned error can't find the container with id d1f866b1693c62835c88b95333e77ef651de3437918975608e3175be38e02820
Sep 16 18:46:14.129029 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:14.128994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648469b4d6-mb9jb" event={"ID":"e67eb7c8-649e-4d76-a7af-9176b7997839","Type":"ContainerStarted","Data":"af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4"}
Sep 16 18:46:14.129400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:14.129051    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648469b4d6-mb9jb" event={"ID":"e67eb7c8-649e-4d76-a7af-9176b7997839","Type":"ContainerStarted","Data":"d1f866b1693c62835c88b95333e77ef651de3437918975608e3175be38e02820"}
Sep 16 18:46:14.151585 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:14.151532    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-648469b4d6-mb9jb" podStartSLOduration=2.15151696 podStartE2EDuration="2.15151696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:14.151229817 +0000 UTC m=+158.263026481" watchObservedRunningTime="2026-09-16 18:46:14.15151696 +0000 UTC m=+158.263313600"
Sep 16 18:46:18.677218 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:18.677168    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:18.801278 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:18.801237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:23.200805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:23.200768    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:23.201206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:23.200843    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:23.205493 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:23.205472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:24.167647 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:24.167617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:46:24.266581 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:24.266552    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8884b7-bzsz2"]
Sep 16 18:46:24.435652 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:24.435576    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:24.545185 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:24.545141    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.229386 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:48.229351    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:48.302763 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:48.302718    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:46:49.287751 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.287717    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f8884b7-bzsz2" podUID="d4624f55-bf0f-4b9e-9118-2caabf792a1e" containerName="console" containerID="cri-o://791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a" gracePeriod=15
Sep 16 18:46:49.526780 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.526759    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:46:49.558658 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-config\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.558658 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-trusted-ca-bundle\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.558843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558675    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-oauth-serving-cert\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.558843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-service-ca\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.558843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558731    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-serving-cert\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.558843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-oauth-config\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.558843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.558810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fzrq\" (UniqueName: \"kubernetes.io/projected/d4624f55-bf0f-4b9e-9118-2caabf792a1e-kube-api-access-6fzrq\") pod \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\" (UID: \"d4624f55-bf0f-4b9e-9118-2caabf792a1e\") "
Sep 16 18:46:49.559169 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.559140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:49.559232 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.559168    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-config" (OuterVolumeSpecName: "console-config") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:49.559286 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.559263    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-service-ca" (OuterVolumeSpecName: "service-ca") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:49.559341 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.559258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:49.560840 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.560817    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4624f55-bf0f-4b9e-9118-2caabf792a1e-kube-api-access-6fzrq" (OuterVolumeSpecName: "kube-api-access-6fzrq") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "kube-api-access-6fzrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:49.561007 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.560849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:49.561007 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.560934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d4624f55-bf0f-4b9e-9118-2caabf792a1e" (UID: "d4624f55-bf0f-4b9e-9118-2caabf792a1e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:49.659351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659317    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fzrq\" (UniqueName: \"kubernetes.io/projected/d4624f55-bf0f-4b9e-9118-2caabf792a1e-kube-api-access-6fzrq\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:49.659351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659351    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:49.659351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659360    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:49.659504 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659369    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:49.659504 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659377    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4624f55-bf0f-4b9e-9118-2caabf792a1e-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:49.659504 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659385    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:49.659504 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:49.659393    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4624f55-bf0f-4b9e-9118-2caabf792a1e-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:46:50.255093 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.255061    2566 generic.go:356] "Generic (PLEG): container finished" podID="d4624f55-bf0f-4b9e-9118-2caabf792a1e" containerID="791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a" exitCode=0
Sep 16 18:46:50.255281 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.255106    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f8884b7-bzsz2"
Sep 16 18:46:50.255281 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.255125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8884b7-bzsz2" event={"ID":"d4624f55-bf0f-4b9e-9118-2caabf792a1e","Type":"ContainerDied","Data":"791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a"}
Sep 16 18:46:50.255281 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.255163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f8884b7-bzsz2" event={"ID":"d4624f55-bf0f-4b9e-9118-2caabf792a1e","Type":"ContainerDied","Data":"fb88f2009113114bd661ec95f6eac6cd366f59eff3e5141420a58d96e0d898f2"}
Sep 16 18:46:50.255281 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.255182    2566 scope.go:117] "RemoveContainer" containerID="791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a"
Sep 16 18:46:50.263519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.263504    2566 scope.go:117] "RemoveContainer" containerID="791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a"
Sep 16 18:46:50.263764 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:46:50.263744    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a\": container with ID starting with 791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a not found: ID does not exist" containerID="791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a"
Sep 16 18:46:50.263829 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.263771    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a"} err="failed to get container status \"791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a\": rpc error: code = NotFound desc = could not find container \"791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a\": container with ID starting with 791b52d380ab74fae72a7f17eddc3539831b031df8cae96c8eeddbdc57fcf60a not found: ID does not exist"
Sep 16 18:46:50.284047 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.284024    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f8884b7-bzsz2"]
Sep 16 18:46:50.293420 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.293399    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f8884b7-bzsz2"]
Sep 16 18:46:50.401421 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:46:50.401395    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4624f55-bf0f-4b9e-9118-2caabf792a1e" path="/var/lib/kubelet/pods/d4624f55-bf0f-4b9e-9118-2caabf792a1e/volumes"
Sep 16 18:47:07.055209 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:07.055170    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:07.153201 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:07.153169    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:07.233505 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:07.233466    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:08.568153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.568123    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"]
Sep 16 18:47:08.568540 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.568367    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4624f55-bf0f-4b9e-9118-2caabf792a1e" containerName="console"
Sep 16 18:47:08.568540 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.568378    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4624f55-bf0f-4b9e-9118-2caabf792a1e" containerName="console"
Sep 16 18:47:08.568540 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.568427    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4624f55-bf0f-4b9e-9118-2caabf792a1e" containerName="console"
Sep 16 18:47:08.570641 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.570624    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.573422 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.573403    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:47:08.573551 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.573496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mprtj\""
Sep 16 18:47:08.574482 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.574466    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:47:08.580914 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.580869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmbd\" (UniqueName: \"kubernetes.io/projected/4057ef36-872f-4438-855f-e3f386a3039f-kube-api-access-lgmbd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.581003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.580939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.581003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.580979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.584378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.584352    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"]
Sep 16 18:47:08.681676 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.681623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.681832 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.681713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmbd\" (UniqueName: \"kubernetes.io/projected/4057ef36-872f-4438-855f-e3f386a3039f-kube-api-access-lgmbd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.681832 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.681747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.682049 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.682028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.682092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.682073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.691470 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.691441    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmbd\" (UniqueName: \"kubernetes.io/projected/4057ef36-872f-4438-855f-e3f386a3039f-kube-api-access-lgmbd\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:08.881426 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:08.881347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:09.166566 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:09.166540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"]
Sep 16 18:47:09.167760 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:47:09.167734    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4057ef36_872f_4438_855f_e3f386a3039f.slice/crio-65174a4bd5d917d0484eddd58d139dd6c1338bec8ab77a648b87b25f2a20f32b WatchSource:0}: Error finding container 65174a4bd5d917d0484eddd58d139dd6c1338bec8ab77a648b87b25f2a20f32b: Status 404 returned error can't find the container with id 65174a4bd5d917d0484eddd58d139dd6c1338bec8ab77a648b87b25f2a20f32b
Sep 16 18:47:09.321911 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:09.321866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g" event={"ID":"4057ef36-872f-4438-855f-e3f386a3039f","Type":"ContainerStarted","Data":"65174a4bd5d917d0484eddd58d139dd6c1338bec8ab77a648b87b25f2a20f32b"}
Sep 16 18:47:16.350104 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:16.350066    2566 generic.go:356] "Generic (PLEG): container finished" podID="4057ef36-872f-4438-855f-e3f386a3039f" containerID="43c4e52680f9bd8131dab9d4b26ed53fc6de4cf9c05f7361dde688aa65485108" exitCode=0
Sep 16 18:47:16.350482 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:16.350162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g" event={"ID":"4057ef36-872f-4438-855f-e3f386a3039f","Type":"ContainerDied","Data":"43c4e52680f9bd8131dab9d4b26ed53fc6de4cf9c05f7361dde688aa65485108"}
Sep 16 18:47:19.365240 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:19.365205    2566 generic.go:356] "Generic (PLEG): container finished" podID="4057ef36-872f-4438-855f-e3f386a3039f" containerID="c030df377b5817f17f233dce2d382d2149a055ad71fec8eab4652186fb1bbc28" exitCode=0
Sep 16 18:47:19.365703 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:19.365250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g" event={"ID":"4057ef36-872f-4438-855f-e3f386a3039f","Type":"ContainerDied","Data":"c030df377b5817f17f233dce2d382d2149a055ad71fec8eab4652186fb1bbc28"}
Sep 16 18:47:26.396349 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:26.396316    2566 generic.go:356] "Generic (PLEG): container finished" podID="4057ef36-872f-4438-855f-e3f386a3039f" containerID="868440881f4c3a82ebb920d5ce2fec6c7d5154299ba7cc78e2cd60a319519edf" exitCode=0
Sep 16 18:47:26.400427 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:26.400402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g" event={"ID":"4057ef36-872f-4438-855f-e3f386a3039f","Type":"ContainerDied","Data":"868440881f4c3a82ebb920d5ce2fec6c7d5154299ba7cc78e2cd60a319519edf"}
Sep 16 18:47:27.519822 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.519797    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:27.634538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.634507    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgmbd\" (UniqueName: \"kubernetes.io/projected/4057ef36-872f-4438-855f-e3f386a3039f-kube-api-access-lgmbd\") pod \"4057ef36-872f-4438-855f-e3f386a3039f\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") "
Sep 16 18:47:27.634729 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.634564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-util\") pod \"4057ef36-872f-4438-855f-e3f386a3039f\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") "
Sep 16 18:47:27.634729 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.634605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-bundle\") pod \"4057ef36-872f-4438-855f-e3f386a3039f\" (UID: \"4057ef36-872f-4438-855f-e3f386a3039f\") "
Sep 16 18:47:27.635270 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.635245    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-bundle" (OuterVolumeSpecName: "bundle") pod "4057ef36-872f-4438-855f-e3f386a3039f" (UID: "4057ef36-872f-4438-855f-e3f386a3039f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:27.636765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.636741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4057ef36-872f-4438-855f-e3f386a3039f-kube-api-access-lgmbd" (OuterVolumeSpecName: "kube-api-access-lgmbd") pod "4057ef36-872f-4438-855f-e3f386a3039f" (UID: "4057ef36-872f-4438-855f-e3f386a3039f"). InnerVolumeSpecName "kube-api-access-lgmbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:27.640496 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.640473    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-util" (OuterVolumeSpecName: "util") pod "4057ef36-872f-4438-855f-e3f386a3039f" (UID: "4057ef36-872f-4438-855f-e3f386a3039f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:27.735543 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.735440    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmbd\" (UniqueName: \"kubernetes.io/projected/4057ef36-872f-4438-855f-e3f386a3039f-kube-api-access-lgmbd\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:47:27.735543 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.735483    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:47:27.735543 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:27.735493    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4057ef36-872f-4438-855f-e3f386a3039f-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:47:28.404524 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:28.404487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g" event={"ID":"4057ef36-872f-4438-855f-e3f386a3039f","Type":"ContainerDied","Data":"65174a4bd5d917d0484eddd58d139dd6c1338bec8ab77a648b87b25f2a20f32b"}
Sep 16 18:47:28.404720 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:28.404540    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65174a4bd5d917d0484eddd58d139dd6c1338bec8ab77a648b87b25f2a20f32b"
Sep 16 18:47:28.404720 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:28.404501    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1asg96g"
Sep 16 18:47:30.091538 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:30.091499    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:30.206774 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:30.206725    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:31.792981 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.792944    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"]
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793204    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="extract"
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793214    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="extract"
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793233    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="util"
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793238    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="util"
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793246    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="pull"
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="pull"
Sep 16 18:47:31.793400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.793292    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4057ef36-872f-4438-855f-e3f386a3039f" containerName="extract"
Sep 16 18:47:31.827353 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.827315    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"]
Sep 16 18:47:31.827525 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.827440    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:31.831609 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.831582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-k2hp9\""
Sep 16 18:47:31.831782 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.831693    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 16 18:47:31.832546 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.832527    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 16 18:47:31.832740 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.832725    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 16 18:47:31.865008 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.864974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkpnq\" (UniqueName: \"kubernetes.io/projected/42ca684c-2f91-457a-866b-1cba10bdcb5e-kube-api-access-kkpnq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-vsg9b\" (UID: \"42ca684c-2f91-457a-866b-1cba10bdcb5e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:31.865179 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.865027    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/42ca684c-2f91-457a-866b-1cba10bdcb5e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-vsg9b\" (UID: \"42ca684c-2f91-457a-866b-1cba10bdcb5e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:31.966350 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.966304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkpnq\" (UniqueName: \"kubernetes.io/projected/42ca684c-2f91-457a-866b-1cba10bdcb5e-kube-api-access-kkpnq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-vsg9b\" (UID: \"42ca684c-2f91-457a-866b-1cba10bdcb5e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:31.966506 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.966362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/42ca684c-2f91-457a-866b-1cba10bdcb5e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-vsg9b\" (UID: \"42ca684c-2f91-457a-866b-1cba10bdcb5e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:31.968578 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.968557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/42ca684c-2f91-457a-866b-1cba10bdcb5e-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-vsg9b\" (UID: \"42ca684c-2f91-457a-866b-1cba10bdcb5e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:31.976655 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:31.976624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkpnq\" (UniqueName: \"kubernetes.io/projected/42ca684c-2f91-457a-866b-1cba10bdcb5e-kube-api-access-kkpnq\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-vsg9b\" (UID: \"42ca684c-2f91-457a-866b-1cba10bdcb5e\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:32.137598 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:32.137560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:32.501466 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:47:32.501440    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ca684c_2f91_457a_866b_1cba10bdcb5e.slice/crio-e4aa21005cae3552e066464e64beb4688ebba3117b5ff2854bbf43d905dd21a3 WatchSource:0}: Error finding container e4aa21005cae3552e066464e64beb4688ebba3117b5ff2854bbf43d905dd21a3: Status 404 returned error can't find the container with id e4aa21005cae3552e066464e64beb4688ebba3117b5ff2854bbf43d905dd21a3
Sep 16 18:47:32.503958 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:32.503929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"]
Sep 16 18:47:33.424076 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:33.424042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b" event={"ID":"42ca684c-2f91-457a-866b-1cba10bdcb5e","Type":"ContainerStarted","Data":"e4aa21005cae3552e066464e64beb4688ebba3117b5ff2854bbf43d905dd21a3"}
Sep 16 18:47:42.175657 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.175618    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:42.457943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.457846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b" event={"ID":"42ca684c-2f91-457a-866b-1cba10bdcb5e","Type":"ContainerStarted","Data":"0c6f63b0760aa9b1ca4e1faaac0e8236d0b1788c98beec75ba3a715b1865e170"}
Sep 16 18:47:42.457943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.457928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:47:42.486708 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.486657    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b" podStartSLOduration=2.289057401 podStartE2EDuration="11.486638801s" totalImagesPullingTime="9.1975814s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:42.484826339 +0000 UTC m=+246.596622992" watchObservedRunningTime="2026-09-16 18:47:42.486638801 +0000 UTC m=+246.598435444"
Sep 16 18:47:42.495937 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.495907    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:42.496169 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.496151    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-bqwm9"]
Sep 16 18:47:42.498293 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.498275    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.503762 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.503740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 16 18:47:42.504102 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.504086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-cbhts\""
Sep 16 18:47:42.504321 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.504307    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 16 18:47:42.552515 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.552485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f259ed08-fff7-40f5-80da-3ba8b5b73af4-cabundle0\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.552649 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.552531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-certificates\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.552649 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.552633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g44d4\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-kube-api-access-g44d4\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.564839 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.564814    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-bqwm9"]
Sep 16 18:47:42.654066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.654037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f259ed08-fff7-40f5-80da-3ba8b5b73af4-cabundle0\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.654236 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.654078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-certificates\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.654236 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.654125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g44d4\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-kube-api-access-g44d4\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.654352 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:42.654271    2566 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Sep 16 18:47:42.654352 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:42.654295    2566 secret.go:281] references non-existent secret key: ca.crt
Sep 16 18:47:42.654352 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:42.654307    2566 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 16 18:47:42.654352 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:42.654322    2566 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-bqwm9: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:47:42.654552 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:42.654411    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-certificates podName:f259ed08-fff7-40f5-80da-3ba8b5b73af4 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.154390942 +0000 UTC m=+247.266187567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-certificates") pod "keda-operator-579c8958f7-bqwm9" (UID: "f259ed08-fff7-40f5-80da-3ba8b5b73af4") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Sep 16 18:47:42.654773 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.654753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/f259ed08-fff7-40f5-80da-3ba8b5b73af4-cabundle0\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.664090 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.664063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g44d4\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-kube-api-access-g44d4\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:42.945316 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:42.945277    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.059378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.059338    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"]
Sep 16 18:47:43.062075 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.062054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.064988 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:43.064957    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"keda-metrics-apiserver-77857b7b98-c8h9b\" is forbidden: User \"system:node:ip-10-0-143-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-143-190.ec2.internal' and this object" podUID="8b935e6c-59f7-499a-81e1-49f0a41ba41c" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.065265 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:47:43.065241    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"keda-metrics-apiserver-certs\" is forbidden: User \"system:node:ip-10-0-143-190.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-keda\": no relationship found between node 'ip-10-0-143-190.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" type="*v1.Secret"
Sep 16 18:47:43.089748 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.089723    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"]
Sep 16 18:47:43.158567 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.158542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8b935e6c-59f7-499a-81e1-49f0a41ba41c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.158725 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.158596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptrf\" (UniqueName: \"kubernetes.io/projected/8b935e6c-59f7-499a-81e1-49f0a41ba41c-kube-api-access-9ptrf\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.158725 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.158669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-certificates\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:43.158725 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.158695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b935e6c-59f7-499a-81e1-49f0a41ba41c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.161018 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.160999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/f259ed08-fff7-40f5-80da-3ba8b5b73af4-certificates\") pod \"keda-operator-579c8958f7-bqwm9\" (UID: \"f259ed08-fff7-40f5-80da-3ba8b5b73af4\") " pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:43.167278 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.167257    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.260076 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.260003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptrf\" (UniqueName: \"kubernetes.io/projected/8b935e6c-59f7-499a-81e1-49f0a41ba41c-kube-api-access-9ptrf\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.260076 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.260067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b935e6c-59f7-499a-81e1-49f0a41ba41c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.260510 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.260107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8b935e6c-59f7-499a-81e1-49f0a41ba41c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.260561 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.260540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/8b935e6c-59f7-499a-81e1-49f0a41ba41c-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.285869 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.285832    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptrf\" (UniqueName: \"kubernetes.io/projected/8b935e6c-59f7-499a-81e1-49f0a41ba41c-kube-api-access-9ptrf\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:43.392969 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.392934    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6qph7"]
Sep 16 18:47:43.395605 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.395588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.402754 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.402732    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 16 18:47:43.410520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.410497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:43.417317 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.417290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.451209 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.451128    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6qph7"]
Sep 16 18:47:43.462016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.461789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ef794b87-1909-40dc-97f5-e9cf3f180260-certificates\") pod \"keda-admission-6c4d864b86-6qph7\" (UID: \"ef794b87-1909-40dc-97f5-e9cf3f180260\") " pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.462016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.461853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkfx7\" (UniqueName: \"kubernetes.io/projected/ef794b87-1909-40dc-97f5-e9cf3f180260-kube-api-access-wkfx7\") pod \"keda-admission-6c4d864b86-6qph7\" (UID: \"ef794b87-1909-40dc-97f5-e9cf3f180260\") " pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.562613 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.562529    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkfx7\" (UniqueName: \"kubernetes.io/projected/ef794b87-1909-40dc-97f5-e9cf3f180260-kube-api-access-wkfx7\") pod \"keda-admission-6c4d864b86-6qph7\" (UID: \"ef794b87-1909-40dc-97f5-e9cf3f180260\") " pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.562761 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.562675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ef794b87-1909-40dc-97f5-e9cf3f180260-certificates\") pod \"keda-admission-6c4d864b86-6qph7\" (UID: \"ef794b87-1909-40dc-97f5-e9cf3f180260\") " pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.565368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.565342    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ef794b87-1909-40dc-97f5-e9cf3f180260-certificates\") pod \"keda-admission-6c4d864b86-6qph7\" (UID: \"ef794b87-1909-40dc-97f5-e9cf3f180260\") " pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.575950 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.575931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkfx7\" (UniqueName: \"kubernetes.io/projected/ef794b87-1909-40dc-97f5-e9cf3f180260-kube-api-access-wkfx7\") pod \"keda-admission-6c4d864b86-6qph7\" (UID: \"ef794b87-1909-40dc-97f5-e9cf3f180260\") " pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.596145 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.596118    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.698655 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.698632    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-bqwm9"]
Sep 16 18:47:43.700916 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:47:43.700891    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf259ed08_fff7_40f5_80da_3ba8b5b73af4.slice/crio-bf0233e2b67251090ab674a6fc7e4557ab8071504675f3ca4e1c48e41e80e513 WatchSource:0}: Error finding container bf0233e2b67251090ab674a6fc7e4557ab8071504675f3ca4e1c48e41e80e513: Status 404 returned error can't find the container with id bf0233e2b67251090ab674a6fc7e4557ab8071504675f3ca4e1c48e41e80e513
Sep 16 18:47:43.705712 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.705693    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:43.909264 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.909220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:43.990910 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:43.990874    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-6qph7"]
Sep 16 18:47:43.992054 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:47:43.992030    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef794b87_1909_40dc_97f5_e9cf3f180260.slice/crio-e9e0e5a999059e85b6d7536dbfce44a6e763cda82f942ecedafe4c2f08028c50 WatchSource:0}: Error finding container e9e0e5a999059e85b6d7536dbfce44a6e763cda82f942ecedafe4c2f08028c50: Status 404 returned error can't find the container with id e9e0e5a999059e85b6d7536dbfce44a6e763cda82f942ecedafe4c2f08028c50
Sep 16 18:47:44.183514 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.183438    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 16 18:47:44.193656 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.193625    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/8b935e6c-59f7-499a-81e1-49f0a41ba41c-certificates\") pod \"keda-metrics-apiserver-77857b7b98-c8h9b\" (UID: \"8b935e6c-59f7-499a-81e1-49f0a41ba41c\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:44.211157 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.211126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.274171 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.274148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:44.452676 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.452581    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:44.469362 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.469330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6qph7" event={"ID":"ef794b87-1909-40dc-97f5-e9cf3f180260","Type":"ContainerStarted","Data":"e9e0e5a999059e85b6d7536dbfce44a6e763cda82f942ecedafe4c2f08028c50"}
Sep 16 18:47:44.470375 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.470354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-bqwm9" event={"ID":"f259ed08-fff7-40f5-80da-3ba8b5b73af4","Type":"ContainerStarted","Data":"bf0233e2b67251090ab674a6fc7e4557ab8071504675f3ca4e1c48e41e80e513"}
Sep 16 18:47:44.560889 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.560666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"]
Sep 16 18:47:44.563336 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:47:44.563306    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b935e6c_59f7_499a_81e1_49f0a41ba41c.slice/crio-958bdaac245546d03bf925fd8571681e82f49599e24f26222efa66c0210278f0 WatchSource:0}: Error finding container 958bdaac245546d03bf925fd8571681e82f49599e24f26222efa66c0210278f0: Status 404 returned error can't find the container with id 958bdaac245546d03bf925fd8571681e82f49599e24f26222efa66c0210278f0
Sep 16 18:47:44.597773 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:44.597742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:47:45.478627 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:45.478587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b" event={"ID":"8b935e6c-59f7-499a-81e1-49f0a41ba41c","Type":"ContainerStarted","Data":"958bdaac245546d03bf925fd8571681e82f49599e24f26222efa66c0210278f0"}
Sep 16 18:47:46.486447 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:46.486408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-6qph7" event={"ID":"ef794b87-1909-40dc-97f5-e9cf3f180260","Type":"ContainerStarted","Data":"b5677bcbb990274f80ddd772d27b986480072e20afe00e00303f5a6fbd8ecf69"}
Sep 16 18:47:46.486946 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:46.486564    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:47:46.508831 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:46.508778    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-6qph7" podStartSLOduration=1.7091068059999999 podStartE2EDuration="3.508761826s" totalImagesPullingTime="1.79965502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:46.508180263 +0000 UTC m=+250.619976908" watchObservedRunningTime="2026-09-16 18:47:46.508761826 +0000 UTC m=+250.620558466"
Sep 16 18:47:48.496269 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:48.496227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-bqwm9" event={"ID":"f259ed08-fff7-40f5-80da-3ba8b5b73af4","Type":"ContainerStarted","Data":"464f81cc1e146ec297f6ea73b93d09fec06c2fae89725984620f76a62f47033b"}
Sep 16 18:47:48.496740 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:48.496463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:47:48.498139 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:48.498106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b" event={"ID":"8b935e6c-59f7-499a-81e1-49f0a41ba41c","Type":"ContainerStarted","Data":"63e61a66264117ab85319dcfdcab8bc294e1d846d291f2f2720e294a673d7706"}
Sep 16 18:47:48.498260 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:48.498175    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:47:48.518447 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:48.518395    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-bqwm9" podStartSLOduration=2.204299333 podStartE2EDuration="6.518379789s" totalImagesPullingTime="4.314080456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:48.516493955 +0000 UTC m=+252.628290598" watchObservedRunningTime="2026-09-16 18:47:48.518379789 +0000 UTC m=+252.630176421"
Sep 16 18:47:48.540403 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:48.540355    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b" podStartSLOduration=2.088598089 podStartE2EDuration="5.54034213s" totalImagesPullingTime="3.451744041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:48.538963425 +0000 UTC m=+252.650760067" watchObservedRunningTime="2026-09-16 18:47:48.54034213 +0000 UTC m=+252.652138772"
Sep 16 18:47:59.507443 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:47:59.507412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-c8h9b"
Sep 16 18:48:03.467199 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:03.467163    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-vsg9b"
Sep 16 18:48:05.959365 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:05.959328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:06.052938 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:06.052905    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:07.493275 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:07.493246    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-6qph7"
Sep 16 18:48:09.505332 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:09.505297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-bqwm9"
Sep 16 18:48:34.106780 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.106743    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:34.856641 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.856609    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"]
Sep 16 18:48:34.859049 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.859033    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:34.864603 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:48:34.864572    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" is forbidden: User \"system:node:ip-10-0-143-190.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-143-190.ec2.internal' and this object" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:34.864603 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.864591    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:48:34.866053 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.866009    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mprtj\""
Sep 16 18:48:34.866053 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.866033    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:48:34.885558 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.885524    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"]
Sep 16 18:48:34.963085 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.963051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:34.963085 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.963086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkfb\" (UniqueName: \"kubernetes.io/projected/b7506f64-2325-48f3-8c79-4cd6d19394ca-kube-api-access-rpkfb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:34.963293 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:34.963112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.064439 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.064400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.064439 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.064448    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkfb\" (UniqueName: \"kubernetes.io/projected/b7506f64-2325-48f3-8c79-4cd6d19394ca-kube-api-access-rpkfb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.064665 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.064474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.064809 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.064787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.064870 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.064817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.074541 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.074521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkfb\" (UniqueName: \"kubernetes.io/projected/b7506f64-2325-48f3-8c79-4cd6d19394ca-kube-api-access-rpkfb\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.168609 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.168578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:35.446045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.446019    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"]
Sep 16 18:48:35.448394 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:48:35.448369    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7506f64_2325_48f3_8c79_4cd6d19394ca.slice/crio-ee1641d8d9ac2062086e78a70ea32724c76b15492a43e13eeaa780371a18c00c WatchSource:0}: Error finding container ee1641d8d9ac2062086e78a70ea32724c76b15492a43e13eeaa780371a18c00c: Status 404 returned error can't find the container with id ee1641d8d9ac2062086e78a70ea32724c76b15492a43e13eeaa780371a18c00c
Sep 16 18:48:35.479440 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.479403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:35.692053 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.692022    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerID="2b9cb65a16de5dbb5b4facd7b9e524f22171f337bc663b88334b18bf2a057fc0" exitCode=0
Sep 16 18:48:35.692053 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.692058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn" event={"ID":"b7506f64-2325-48f3-8c79-4cd6d19394ca","Type":"ContainerDied","Data":"2b9cb65a16de5dbb5b4facd7b9e524f22171f337bc663b88334b18bf2a057fc0"}
Sep 16 18:48:35.692376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:35.692091    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn" event={"ID":"b7506f64-2325-48f3-8c79-4cd6d19394ca","Type":"ContainerStarted","Data":"ee1641d8d9ac2062086e78a70ea32724c76b15492a43e13eeaa780371a18c00c"}
Sep 16 18:48:36.293901 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:36.293859    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:48:37.707709 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:37.707676    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerID="e278f7fd9b6a6d1498353e1e73ee83799faf1e16a71d6612603655f88ec6c9d9" exitCode=0
Sep 16 18:48:37.708135 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:37.707714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn" event={"ID":"b7506f64-2325-48f3-8c79-4cd6d19394ca","Type":"ContainerDied","Data":"e278f7fd9b6a6d1498353e1e73ee83799faf1e16a71d6612603655f88ec6c9d9"}
Sep 16 18:48:38.714441 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:38.714410    2566 generic.go:356] "Generic (PLEG): container finished" podID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerID="688f59a4a76a621bef376733faae9079bd4f5c67dc0a7a227308595a8878224a" exitCode=0
Sep 16 18:48:38.714816 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:38.714448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn" event={"ID":"b7506f64-2325-48f3-8c79-4cd6d19394ca","Type":"ContainerDied","Data":"688f59a4a76a621bef376733faae9079bd4f5c67dc0a7a227308595a8878224a"}
Sep 16 18:48:39.843044 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.843017    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:39.905650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.905623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-util\") pod \"b7506f64-2325-48f3-8c79-4cd6d19394ca\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") "
Sep 16 18:48:39.905790 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.905662    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpkfb\" (UniqueName: \"kubernetes.io/projected/b7506f64-2325-48f3-8c79-4cd6d19394ca-kube-api-access-rpkfb\") pod \"b7506f64-2325-48f3-8c79-4cd6d19394ca\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") "
Sep 16 18:48:39.905790 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.905718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-bundle\") pod \"b7506f64-2325-48f3-8c79-4cd6d19394ca\" (UID: \"b7506f64-2325-48f3-8c79-4cd6d19394ca\") "
Sep 16 18:48:39.906445 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.906421    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-bundle" (OuterVolumeSpecName: "bundle") pod "b7506f64-2325-48f3-8c79-4cd6d19394ca" (UID: "b7506f64-2325-48f3-8c79-4cd6d19394ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:39.907828 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.907801    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7506f64-2325-48f3-8c79-4cd6d19394ca-kube-api-access-rpkfb" (OuterVolumeSpecName: "kube-api-access-rpkfb") pod "b7506f64-2325-48f3-8c79-4cd6d19394ca" (UID: "b7506f64-2325-48f3-8c79-4cd6d19394ca"). InnerVolumeSpecName "kube-api-access-rpkfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:48:39.913730 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:39.913707    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-util" (OuterVolumeSpecName: "util") pod "b7506f64-2325-48f3-8c79-4cd6d19394ca" (UID: "b7506f64-2325-48f3-8c79-4cd6d19394ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:48:40.006798 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:40.006724    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:40.006798 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:40.006750    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkfb\" (UniqueName: \"kubernetes.io/projected/b7506f64-2325-48f3-8c79-4cd6d19394ca-kube-api-access-rpkfb\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:40.006798 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:40.006761    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b7506f64-2325-48f3-8c79-4cd6d19394ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:48:40.725503 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:40.725426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn"
Sep 16 18:48:40.725643 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:40.725423    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bbvzn" event={"ID":"b7506f64-2325-48f3-8c79-4cd6d19394ca","Type":"ContainerDied","Data":"ee1641d8d9ac2062086e78a70ea32724c76b15492a43e13eeaa780371a18c00c"}
Sep 16 18:48:40.725643 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:40.725552    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee1641d8d9ac2062086e78a70ea32724c76b15492a43e13eeaa780371a18c00c"
Sep 16 18:48:45.998665 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:45.998632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:47.318509 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.318479    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:47.940755 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.940722    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"]
Sep 16 18:48:47.941038 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941022    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="extract"
Sep 16 18:48:47.941097 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941040    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="extract"
Sep 16 18:48:47.941097 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941061    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="pull"
Sep 16 18:48:47.941097 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941066    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="pull"
Sep 16 18:48:47.941097 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941072    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="util"
Sep 16 18:48:47.941097 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941078    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="util"
Sep 16 18:48:47.941261 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.941140    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7506f64-2325-48f3-8c79-4cd6d19394ca" containerName="extract"
Sep 16 18:48:47.943223 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.943208    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:47.946164 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.946139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-fskzs\""
Sep 16 18:48:47.946289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.946234    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 16 18:48:47.946289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.946252    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:48:47.956560 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:47.956540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"]
Sep 16 18:48:48.068010 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.067987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf673a77-084e-442f-8caa-38522b5c3245-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hgbzg\" (UID: \"bf673a77-084e-442f-8caa-38522b5c3245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.068143 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.068044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5qf\" (UniqueName: \"kubernetes.io/projected/bf673a77-084e-442f-8caa-38522b5c3245-kube-api-access-hh5qf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hgbzg\" (UID: \"bf673a77-084e-442f-8caa-38522b5c3245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.168624 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.168598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5qf\" (UniqueName: \"kubernetes.io/projected/bf673a77-084e-442f-8caa-38522b5c3245-kube-api-access-hh5qf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hgbzg\" (UID: \"bf673a77-084e-442f-8caa-38522b5c3245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.168719 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.168637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf673a77-084e-442f-8caa-38522b5c3245-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hgbzg\" (UID: \"bf673a77-084e-442f-8caa-38522b5c3245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.168977 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.168962    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf673a77-084e-442f-8caa-38522b5c3245-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hgbzg\" (UID: \"bf673a77-084e-442f-8caa-38522b5c3245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.178068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.178045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5qf\" (UniqueName: \"kubernetes.io/projected/bf673a77-084e-442f-8caa-38522b5c3245-kube-api-access-hh5qf\") pod \"cert-manager-operator-controller-manager-684f79c9b9-hgbzg\" (UID: \"bf673a77-084e-442f-8caa-38522b5c3245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.251849 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.251791    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"
Sep 16 18:48:48.528752 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.528574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg"]
Sep 16 18:48:48.531702 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:48:48.531676    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf673a77_084e_442f_8caa_38522b5c3245.slice/crio-e3be673654beaa78769b5c4dce0071e08936fb3085cdfe4295b058f4af48a346 WatchSource:0}: Error finding container e3be673654beaa78769b5c4dce0071e08936fb3085cdfe4295b058f4af48a346: Status 404 returned error can't find the container with id e3be673654beaa78769b5c4dce0071e08936fb3085cdfe4295b058f4af48a346
Sep 16 18:48:48.534437 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.534419    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:48:48.765419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:48.765387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg" event={"ID":"bf673a77-084e-442f-8caa-38522b5c3245","Type":"ContainerStarted","Data":"e3be673654beaa78769b5c4dce0071e08936fb3085cdfe4295b058f4af48a346"}
Sep 16 18:48:49.809978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:49.809939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:49.910271 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:49.910233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:50.780256 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:50.780218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg" event={"ID":"bf673a77-084e-442f-8caa-38522b5c3245","Type":"ContainerStarted","Data":"81636c5df2a04ef9d68db6217d92da9d5a8c69d2f4a8df8d9f11e12b735d06a6"}
Sep 16 18:48:50.812507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:50.812454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-hgbzg" podStartSLOduration=2.41331621 podStartE2EDuration="3.812439205s" totalImagesPullingTime="1.399122995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:50.811946708 +0000 UTC m=+314.923743350" watchObservedRunningTime="2026-09-16 18:48:50.812439205 +0000 UTC m=+314.924235846"
Sep 16 18:48:51.770815 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:51.770772    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:51.904236 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:51.904196    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.164406 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:54.164362    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.374138 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:54.374075    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.802302 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:54.802261    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:54.867654 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:54.867624    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:55.576256 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:55.576222    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:55.706104 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:55.706069    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:55.809832 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:55.809797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:48:56.429494 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.429446    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"]
Sep 16 18:48:56.432013 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.431998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.434694 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.434667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:48:56.434834 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.434699    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:48:56.435758 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.435736    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mprtj\""
Sep 16 18:48:56.441857 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.441835    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"]
Sep 16 18:48:56.541708 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.541671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.541858 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.541747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.541957 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.541860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn2ph\" (UniqueName: \"kubernetes.io/projected/dd66eba2-657c-4e17-b3a1-b0d7d912384f-kube-api-access-fn2ph\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.642962 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.642918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.643415 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.642976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn2ph\" (UniqueName: \"kubernetes.io/projected/dd66eba2-657c-4e17-b3a1-b0d7d912384f-kube-api-access-fn2ph\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.643415 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.643030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.643415 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.643264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.643546 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.643417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.652467 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.652440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn2ph\" (UniqueName: \"kubernetes.io/projected/dd66eba2-657c-4e17-b3a1-b0d7d912384f-kube-api-access-fn2ph\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:56.741158 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:56.741079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:48:57.026578 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:57.026540    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"]
Sep 16 18:48:57.028704 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:48:57.028677    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd66eba2_657c_4e17_b3a1_b0d7d912384f.slice/crio-b1a84f4e2957e9f4e88a9e984873cabb53ebcda95d56b74fad38e12f7bc88ffc WatchSource:0}: Error finding container b1a84f4e2957e9f4e88a9e984873cabb53ebcda95d56b74fad38e12f7bc88ffc: Status 404 returned error can't find the container with id b1a84f4e2957e9f4e88a9e984873cabb53ebcda95d56b74fad38e12f7bc88ffc
Sep 16 18:48:57.813113 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:57.813084    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerID="a89876d00565bfa9361d7d7caba19cddd3ee2eb7c0f482768425cdfd876d2ff1" exitCode=0
Sep 16 18:48:57.813506 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:57.813183    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w" event={"ID":"dd66eba2-657c-4e17-b3a1-b0d7d912384f","Type":"ContainerDied","Data":"a89876d00565bfa9361d7d7caba19cddd3ee2eb7c0f482768425cdfd876d2ff1"}
Sep 16 18:48:57.813506 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:48:57.813234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w" event={"ID":"dd66eba2-657c-4e17-b3a1-b0d7d912384f","Type":"ContainerStarted","Data":"b1a84f4e2957e9f4e88a9e984873cabb53ebcda95d56b74fad38e12f7bc88ffc"}
Sep 16 18:49:00.830141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:00.830106    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerID="97baae5210df2ae6facbd8df9ccfa41baade524650728a06aa61c0882b2e73f6" exitCode=0
Sep 16 18:49:00.830501 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:00.830169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w" event={"ID":"dd66eba2-657c-4e17-b3a1-b0d7d912384f","Type":"ContainerDied","Data":"97baae5210df2ae6facbd8df9ccfa41baade524650728a06aa61c0882b2e73f6"}
Sep 16 18:49:01.805650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:01.805610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:01.836373 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:01.836338    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerID="410bca957b8038faaa317e6271ffef17c21f4351a93ccb8dbb980afe91d83d80" exitCode=0
Sep 16 18:49:01.836749 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:01.836391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w" event={"ID":"dd66eba2-657c-4e17-b3a1-b0d7d912384f","Type":"ContainerDied","Data":"410bca957b8038faaa317e6271ffef17c21f4351a93ccb8dbb980afe91d83d80"}
Sep 16 18:49:02.133772 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.133673    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh7lm"]
Sep 16 18:49:02.137986 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.137960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.140834 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.140808    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vphd7\""
Sep 16 18:49:02.140962 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.140889    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 16 18:49:02.141419 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.141404    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 16 18:49:02.148650 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.148629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh7lm"]
Sep 16 18:49:02.289310 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.289277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/337b1173-f9f0-431b-a0a9-2353297da3ac-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh7lm\" (UID: \"337b1173-f9f0-431b-a0a9-2353297da3ac\") " pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.289457 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.289331    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jzrn\" (UniqueName: \"kubernetes.io/projected/337b1173-f9f0-431b-a0a9-2353297da3ac-kube-api-access-7jzrn\") pod \"cert-manager-55b4d954d5-kh7lm\" (UID: \"337b1173-f9f0-431b-a0a9-2353297da3ac\") " pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.390553 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.390526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/337b1173-f9f0-431b-a0a9-2353297da3ac-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh7lm\" (UID: \"337b1173-f9f0-431b-a0a9-2353297da3ac\") " pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.390703 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.390571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jzrn\" (UniqueName: \"kubernetes.io/projected/337b1173-f9f0-431b-a0a9-2353297da3ac-kube-api-access-7jzrn\") pod \"cert-manager-55b4d954d5-kh7lm\" (UID: \"337b1173-f9f0-431b-a0a9-2353297da3ac\") " pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.405375 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.405346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jzrn\" (UniqueName: \"kubernetes.io/projected/337b1173-f9f0-431b-a0a9-2353297da3ac-kube-api-access-7jzrn\") pod \"cert-manager-55b4d954d5-kh7lm\" (UID: \"337b1173-f9f0-431b-a0a9-2353297da3ac\") " pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.405555 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.405537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/337b1173-f9f0-431b-a0a9-2353297da3ac-bound-sa-token\") pod \"cert-manager-55b4d954d5-kh7lm\" (UID: \"337b1173-f9f0-431b-a0a9-2353297da3ac\") " pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.455553 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.455527    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-kh7lm"
Sep 16 18:49:02.728521 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.728491    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-kh7lm"]
Sep 16 18:49:02.841927 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.841895    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kh7lm" event={"ID":"337b1173-f9f0-431b-a0a9-2353297da3ac","Type":"ContainerStarted","Data":"87e39cf24c10453d39a52069940ad05cd1c94299d8feb2bf8986f3f236f4b87b"}
Sep 16 18:49:02.955231 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:02.955207    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:49:03.097327 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.097255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fn2ph\" (UniqueName: \"kubernetes.io/projected/dd66eba2-657c-4e17-b3a1-b0d7d912384f-kube-api-access-fn2ph\") pod \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") "
Sep 16 18:49:03.097448 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.097354    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-util\") pod \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") "
Sep 16 18:49:03.097448 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.097390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-bundle\") pod \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\" (UID: \"dd66eba2-657c-4e17-b3a1-b0d7d912384f\") "
Sep 16 18:49:03.097786 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.097759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-bundle" (OuterVolumeSpecName: "bundle") pod "dd66eba2-657c-4e17-b3a1-b0d7d912384f" (UID: "dd66eba2-657c-4e17-b3a1-b0d7d912384f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.099228 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.099204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd66eba2-657c-4e17-b3a1-b0d7d912384f-kube-api-access-fn2ph" (OuterVolumeSpecName: "kube-api-access-fn2ph") pod "dd66eba2-657c-4e17-b3a1-b0d7d912384f" (UID: "dd66eba2-657c-4e17-b3a1-b0d7d912384f"). InnerVolumeSpecName "kube-api-access-fn2ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:03.101832 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.101812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-util" (OuterVolumeSpecName: "util") pod "dd66eba2-657c-4e17-b3a1-b0d7d912384f" (UID: "dd66eba2-657c-4e17-b3a1-b0d7d912384f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:03.198112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.198082    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.198112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.198115    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd66eba2-657c-4e17-b3a1-b0d7d912384f-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.198298 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.198132    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn2ph\" (UniqueName: \"kubernetes.io/projected/dd66eba2-657c-4e17-b3a1-b0d7d912384f-kube-api-access-fn2ph\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:03.849621 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.849542    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w"
Sep 16 18:49:03.850065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.849548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fwp48w" event={"ID":"dd66eba2-657c-4e17-b3a1-b0d7d912384f","Type":"ContainerDied","Data":"b1a84f4e2957e9f4e88a9e984873cabb53ebcda95d56b74fad38e12f7bc88ffc"}
Sep 16 18:49:03.850065 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:03.849671    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1a84f4e2957e9f4e88a9e984873cabb53ebcda95d56b74fad38e12f7bc88ffc"
Sep 16 18:49:05.860400 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:05.860364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-kh7lm" event={"ID":"337b1173-f9f0-431b-a0a9-2353297da3ac","Type":"ContainerStarted","Data":"85ec677b3e48fe662214ded0ba21843e70841eb1ad14f1d5629910b1203964a7"}
Sep 16 18:49:05.877319 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:05.877274    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-kh7lm" podStartSLOduration=1.382410369 podStartE2EDuration="3.87725717s" totalImagesPullingTime="2.494846801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:05.87491326 +0000 UTC m=+329.986709903" watchObservedRunningTime="2026-09-16 18:49:05.87725717 +0000 UTC m=+329.989053811"
Sep 16 18:49:08.502068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:08.502028    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:08.599281 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:08.599243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:19.285320 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:19.285282    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:19.385347 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:19.385304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:19.485563 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:19.485529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:21.946307 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:21.946268    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:24.040193 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.040159    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:24.048379 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048355    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"]
Sep 16 18:49:24.048675 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048661    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="util"
Sep 16 18:49:24.048718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048677    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="util"
Sep 16 18:49:24.048718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048684    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="pull"
Sep 16 18:49:24.048718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048690    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="pull"
Sep 16 18:49:24.048718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048701    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="extract"
Sep 16 18:49:24.048718 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048709    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="extract"
Sep 16 18:49:24.048870 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.048757    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd66eba2-657c-4e17-b3a1-b0d7d912384f" containerName="extract"
Sep 16 18:49:24.055117 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.055100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.057808 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.057787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-5c5g4\""
Sep 16 18:49:24.057973 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.057840    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 16 18:49:24.057973 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.057869    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 16 18:49:24.057973 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.057937    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 16 18:49:24.057973 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.057852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 16 18:49:24.058193 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.058167    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 16 18:49:24.058193 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.058181    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 16 18:49:24.075475 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.075456    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"]
Sep 16 18:49:24.116024 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.116002    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:24.158768 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lblq5\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-kube-api-access-lblq5\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.158855 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.158855 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.158952 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.158952 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.158952 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.158952 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.158950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259677 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259805 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259986 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lblq5\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-kube-api-access-lblq5\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259986 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.259986 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.259865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.260425 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.260402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.262120 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.262099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.262205 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.262157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.262416 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.262401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.262533 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.262509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.268284 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.268265    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.268529 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.268511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lblq5\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-kube-api-access-lblq5\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8pl\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.364663 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.364580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:24.640499 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.640471    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"]
Sep 16 18:49:24.642873 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:49:24.642842    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c9e23d7_f682_4d9b_9ca5_4b9650fb1e90.slice/crio-fd601fc23b088dc8e653c9ff76436a72d557f6fa315bf5917a162ba2befb0ec8 WatchSource:0}: Error finding container fd601fc23b088dc8e653c9ff76436a72d557f6fa315bf5917a162ba2befb0ec8: Status 404 returned error can't find the container with id fd601fc23b088dc8e653c9ff76436a72d557f6fa315bf5917a162ba2befb0ec8
Sep 16 18:49:24.942362 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:24.942277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" event={"ID":"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90","Type":"ContainerStarted","Data":"fd601fc23b088dc8e653c9ff76436a72d557f6fa315bf5917a162ba2befb0ec8"}
Sep 16 18:49:25.122127 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:25.122092    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:28.277082 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:28.277042    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:49:28.277461 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:28.277120    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:49:28.963568 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:28.963530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" event={"ID":"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90","Type":"ContainerStarted","Data":"5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223"}
Sep 16 18:49:28.963747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:28.963646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:28.987369 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:28.987313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" podStartSLOduration=1.355564765 podStartE2EDuration="4.987294081s" totalImagesPullingTime="3.631729316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:28.983837142 +0000 UTC m=+353.095633783" watchObservedRunningTime="2026-09-16 18:49:28.987294081 +0000 UTC m=+353.099090723"
Sep 16 18:49:29.266033 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:29.265937    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:29.325680 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:29.325626    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:29.971077 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:29.971052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:49:30.682323 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:30.682285    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:30.784550 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:30.784517    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:33.684384 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:33.684346    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.038480 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:37.038445    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:37.160868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:37.160830    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:44.849341 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.849308    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bdc6bbd-p7q24"]
Sep 16 18:49:44.852079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.852059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.868695 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.868671    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc6bbd-p7q24"]
Sep 16 18:49:44.934935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.934901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxzp5\" (UniqueName: \"kubernetes.io/projected/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-kube-api-access-vxzp5\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.935084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.934941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-serving-cert\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.935084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.934958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-trusted-ca-bundle\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.935084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.935003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-oauth-serving-cert\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.935084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.935028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-config\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.935084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.935067    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-oauth-config\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.935275 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.935145    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-service-ca\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:44.952834 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:44.952804    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.036137 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-service-ca\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.036278 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxzp5\" (UniqueName: \"kubernetes.io/projected/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-kube-api-access-vxzp5\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.036278 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-serving-cert\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.036278 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-trusted-ca-bundle\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.036447 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-oauth-serving-cert\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.036447 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-config\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.036534 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.036453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-oauth-config\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.037136 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.037098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-service-ca\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.037136 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.037115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-oauth-serving-cert\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.037317 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.037138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-config\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.037317 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.037167    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-trusted-ca-bundle\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.038770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.038751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-serving-cert\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.038868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.038777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-console-oauth-config\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.046835 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.046810    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxzp5\" (UniqueName: \"kubernetes.io/projected/d0e36b21-a5e2-4100-b41a-9ab8e66a79b0-kube-api-access-vxzp5\") pod \"console-bdc6bbd-p7q24\" (UID: \"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0\") " pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.063604 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.063570    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:45.161688 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.161664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:45.434847 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:45.434820    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bdc6bbd-p7q24"]
Sep 16 18:49:45.436491 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:49:45.436467    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e36b21_a5e2_4100_b41a_9ab8e66a79b0.slice/crio-e14601c35b178e705132aba115fb359881aa3d22556d62b428f247b8891f2ec2 WatchSource:0}: Error finding container e14601c35b178e705132aba115fb359881aa3d22556d62b428f247b8891f2ec2: Status 404 returned error can't find the container with id e14601c35b178e705132aba115fb359881aa3d22556d62b428f247b8891f2ec2
Sep 16 18:49:46.041148 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:46.041109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc6bbd-p7q24" event={"ID":"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0","Type":"ContainerStarted","Data":"653017bce1487cc7e7d5d1820731faefb0a1f82c2d34be24b34e790c3d29df13"}
Sep 16 18:49:46.041533 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:46.041203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bdc6bbd-p7q24" event={"ID":"d0e36b21-a5e2-4100-b41a-9ab8e66a79b0","Type":"ContainerStarted","Data":"e14601c35b178e705132aba115fb359881aa3d22556d62b428f247b8891f2ec2"}
Sep 16 18:49:46.068781 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:46.068738    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bdc6bbd-p7q24" podStartSLOduration=2.068723813 podStartE2EDuration="2.068723813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:46.064038812 +0000 UTC m=+370.175835463" watchObservedRunningTime="2026-09-16 18:49:46.068723813 +0000 UTC m=+370.180520453"
Sep 16 18:49:46.864861 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:46.864828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:47.102548 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:47.102518    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:52.102957 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.102916    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"]
Sep 16 18:49:52.105579 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.105559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.109399 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.109379    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 16 18:49:52.110109 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.110090    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 16 18:49:52.110196 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.110096    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mprtj\""
Sep 16 18:49:52.116066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.116043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"]
Sep 16 18:49:52.194237 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.194202    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"]
Sep 16 18:49:52.196836 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.196819    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.201711 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.201685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.201873 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.201717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpgzj\" (UniqueName: \"kubernetes.io/projected/400b28fe-517b-4d2e-ad01-fb5a61ad4576-kube-api-access-mpgzj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.201873 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.201765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.209063 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.209035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"]
Sep 16 18:49:52.302898 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.302852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.303099 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.302966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6r9m\" (UniqueName: \"kubernetes.io/projected/86d880d4-060f-4ab6-949a-5841d818bbb0-kube-api-access-d6r9m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.303099 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.302992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.303099 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.303026    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.303099 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.303051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpgzj\" (UniqueName: \"kubernetes.io/projected/400b28fe-517b-4d2e-ad01-fb5a61ad4576-kube-api-access-mpgzj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.303313 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.303158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.303417 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.303394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.303493 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.303472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.303741 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.303720    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"]
Sep 16 18:49:52.306469 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.306449    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.313340 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.313317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpgzj\" (UniqueName: \"kubernetes.io/projected/400b28fe-517b-4d2e-ad01-fb5a61ad4576-kube-api-access-mpgzj\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.320117 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.320093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"]
Sep 16 18:49:52.402780 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.402746    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"]
Sep 16 18:49:52.404365 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.404483 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.404483 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404408    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6r9m\" (UniqueName: \"kubernetes.io/projected/86d880d4-060f-4ab6-949a-5841d818bbb0-kube-api-access-d6r9m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.404483 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.404584 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.404622 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9wxv\" (UniqueName: \"kubernetes.io/projected/a846c00b-df9a-4e44-b8be-b76f7146ca48-kube-api-access-b9wxv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.404731 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.404860 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.404843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.405676 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.405655    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.415050 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.415029    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:52.415426 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.415401    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"]
Sep 16 18:49:52.417428 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.417114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6r9m\" (UniqueName: \"kubernetes.io/projected/86d880d4-060f-4ab6-949a-5841d818bbb0-kube-api-access-d6r9m\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.505721 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.505692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:52.505926 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.505874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.506086 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.506205 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5xp\" (UniqueName: \"kubernetes.io/projected/22717e3d-e8b1-4689-a919-a838543238ef-kube-api-access-9s5xp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.506205 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.506205 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.506379 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9wxv\" (UniqueName: \"kubernetes.io/projected/a846c00b-df9a-4e44-b8be-b76f7146ca48-kube-api-access-b9wxv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.506436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.506479 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.506430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.517254 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.517219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9wxv\" (UniqueName: \"kubernetes.io/projected/a846c00b-df9a-4e44-b8be-b76f7146ca48-kube-api-access-b9wxv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.607683 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.607644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5xp\" (UniqueName: \"kubernetes.io/projected/22717e3d-e8b1-4689-a919-a838543238ef-kube-api-access-9s5xp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.607836 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.607689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.607836 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.607718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.608115 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.608097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.608195 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.608176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.616738 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.616721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5xp\" (UniqueName: \"kubernetes.io/projected/22717e3d-e8b1-4689-a919-a838543238ef-kube-api-access-9s5xp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.622562 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.622545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:52.704176 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.704130    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"]
Sep 16 18:49:52.706560 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:49:52.706532    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400b28fe_517b_4d2e_ad01_fb5a61ad4576.slice/crio-72eec2b7b39f644332d085c0aa0f57ccff4654c43d5d489a32841c15a3d63da7 WatchSource:0}: Error finding container 72eec2b7b39f644332d085c0aa0f57ccff4654c43d5d489a32841c15a3d63da7: Status 404 returned error can't find the container with id 72eec2b7b39f644332d085c0aa0f57ccff4654c43d5d489a32841c15a3d63da7
Sep 16 18:49:52.715290 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.715271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:52.794757 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.794729    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"]
Sep 16 18:49:52.798278 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:49:52.798251    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d880d4_060f_4ab6_949a_5841d818bbb0.slice/crio-5ea3551c6dd7f6d71029d7bfc77d37e170627e542469282b07dd9bed986ff6d9 WatchSource:0}: Error finding container 5ea3551c6dd7f6d71029d7bfc77d37e170627e542469282b07dd9bed986ff6d9: Status 404 returned error can't find the container with id 5ea3551c6dd7f6d71029d7bfc77d37e170627e542469282b07dd9bed986ff6d9
Sep 16 18:49:52.898408 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.898384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"]
Sep 16 18:49:52.899484 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:49:52.899458    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda846c00b_df9a_4e44_b8be_b76f7146ca48.slice/crio-1d3dcaf1d0496364a5eea4ffc66339682605407b3d4b99bb93964212fd9b49e1 WatchSource:0}: Error finding container 1d3dcaf1d0496364a5eea4ffc66339682605407b3d4b99bb93964212fd9b49e1: Status 404 returned error can't find the container with id 1d3dcaf1d0496364a5eea4ffc66339682605407b3d4b99bb93964212fd9b49e1
Sep 16 18:49:52.992009 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:52.991988    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"]
Sep 16 18:49:53.027248 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:49:53.027213    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22717e3d_e8b1_4689_a919_a838543238ef.slice/crio-42f8724ba9c2a08158eebb0ffc03b6e0bed6a8b25f1ac5f6979e649ecce2b69e WatchSource:0}: Error finding container 42f8724ba9c2a08158eebb0ffc03b6e0bed6a8b25f1ac5f6979e649ecce2b69e: Status 404 returned error can't find the container with id 42f8724ba9c2a08158eebb0ffc03b6e0bed6a8b25f1ac5f6979e649ecce2b69e
Sep 16 18:49:53.071279 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.071247    2566 generic.go:356] "Generic (PLEG): container finished" podID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerID="1cf8d468ebe0321df565b67bdd031638a20c5433ad3f33c87686b700bca0eaa6" exitCode=0
Sep 16 18:49:53.071442 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.071331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c" event={"ID":"400b28fe-517b-4d2e-ad01-fb5a61ad4576","Type":"ContainerDied","Data":"1cf8d468ebe0321df565b67bdd031638a20c5433ad3f33c87686b700bca0eaa6"}
Sep 16 18:49:53.071442 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.071389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c" event={"ID":"400b28fe-517b-4d2e-ad01-fb5a61ad4576","Type":"ContainerStarted","Data":"72eec2b7b39f644332d085c0aa0f57ccff4654c43d5d489a32841c15a3d63da7"}
Sep 16 18:49:53.072611 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.072589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt" event={"ID":"22717e3d-e8b1-4689-a919-a838543238ef","Type":"ContainerStarted","Data":"42f8724ba9c2a08158eebb0ffc03b6e0bed6a8b25f1ac5f6979e649ecce2b69e"}
Sep 16 18:49:53.074260 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.074239    2566 generic.go:356] "Generic (PLEG): container finished" podID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerID="a730c29123d23279c0028e12823e7ffbae1cce5c5509015bd46bfd607c394c50" exitCode=0
Sep 16 18:49:53.074363 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.074313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx" event={"ID":"a846c00b-df9a-4e44-b8be-b76f7146ca48","Type":"ContainerDied","Data":"a730c29123d23279c0028e12823e7ffbae1cce5c5509015bd46bfd607c394c50"}
Sep 16 18:49:53.074363 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.074348    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx" event={"ID":"a846c00b-df9a-4e44-b8be-b76f7146ca48","Type":"ContainerStarted","Data":"1d3dcaf1d0496364a5eea4ffc66339682605407b3d4b99bb93964212fd9b49e1"}
Sep 16 18:49:53.076079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.076056    2566 generic.go:356] "Generic (PLEG): container finished" podID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerID="eed1f1d3906582b148043404a7459084d1b2a59ea97c90f009b06cc0556828d1" exitCode=0
Sep 16 18:49:53.076172 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.076142    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz" event={"ID":"86d880d4-060f-4ab6-949a-5841d818bbb0","Type":"ContainerDied","Data":"eed1f1d3906582b148043404a7459084d1b2a59ea97c90f009b06cc0556828d1"}
Sep 16 18:49:53.076229 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:53.076179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz" event={"ID":"86d880d4-060f-4ab6-949a-5841d818bbb0","Type":"ContainerStarted","Data":"5ea3551c6dd7f6d71029d7bfc77d37e170627e542469282b07dd9bed986ff6d9"}
Sep 16 18:49:54.084396 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:54.084359    2566 generic.go:356] "Generic (PLEG): container finished" podID="22717e3d-e8b1-4689-a919-a838543238ef" containerID="13012bdbcbb83bbee8d5246968c31eba25758c86cc6086ab928fbe430bb40c3d" exitCode=0
Sep 16 18:49:54.084808 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:54.084448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt" event={"ID":"22717e3d-e8b1-4689-a919-a838543238ef","Type":"ContainerDied","Data":"13012bdbcbb83bbee8d5246968c31eba25758c86cc6086ab928fbe430bb40c3d"}
Sep 16 18:49:54.087067 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:54.086996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx" event={"ID":"a846c00b-df9a-4e44-b8be-b76f7146ca48","Type":"ContainerStarted","Data":"d57bd90105edcf2e2307387bc5ca6e952e499d392bd3f42266284b5196a3ad63"}
Sep 16 18:49:54.089257 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:54.089233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz" event={"ID":"86d880d4-060f-4ab6-949a-5841d818bbb0","Type":"ContainerStarted","Data":"23f36c88af3ddd675b5fc9e942c51c636d4ea75fa04412ba06be0bc6ed4e5004"}
Sep 16 18:49:55.096506 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.096421    2566 generic.go:356] "Generic (PLEG): container finished" podID="22717e3d-e8b1-4689-a919-a838543238ef" containerID="8881105842b1ac461d127e4229cacbd2cba42d1050224dc8e2c7669ce36a2334" exitCode=0
Sep 16 18:49:55.096918 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.096498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt" event={"ID":"22717e3d-e8b1-4689-a919-a838543238ef","Type":"ContainerDied","Data":"8881105842b1ac461d127e4229cacbd2cba42d1050224dc8e2c7669ce36a2334"}
Sep 16 18:49:55.098565 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.098543    2566 generic.go:356] "Generic (PLEG): container finished" podID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerID="d57bd90105edcf2e2307387bc5ca6e952e499d392bd3f42266284b5196a3ad63" exitCode=0
Sep 16 18:49:55.098645 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.098626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx" event={"ID":"a846c00b-df9a-4e44-b8be-b76f7146ca48","Type":"ContainerDied","Data":"d57bd90105edcf2e2307387bc5ca6e952e499d392bd3f42266284b5196a3ad63"}
Sep 16 18:49:55.103388 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.103361    2566 generic.go:356] "Generic (PLEG): container finished" podID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerID="23f36c88af3ddd675b5fc9e942c51c636d4ea75fa04412ba06be0bc6ed4e5004" exitCode=0
Sep 16 18:49:55.103388 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.103387    2566 generic.go:356] "Generic (PLEG): container finished" podID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerID="be97f816119659b39c6889788ddb506d812b82d5ef33f936344281a32cfdd654" exitCode=0
Sep 16 18:49:55.103541 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.103407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz" event={"ID":"86d880d4-060f-4ab6-949a-5841d818bbb0","Type":"ContainerDied","Data":"23f36c88af3ddd675b5fc9e942c51c636d4ea75fa04412ba06be0bc6ed4e5004"}
Sep 16 18:49:55.103541 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.103458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz" event={"ID":"86d880d4-060f-4ab6-949a-5841d818bbb0","Type":"ContainerDied","Data":"be97f816119659b39c6889788ddb506d812b82d5ef33f936344281a32cfdd654"}
Sep 16 18:49:55.162296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.162262    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:55.162410 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.162321    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:55.169291 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:55.169260    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:56.110519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.110481    2566 generic.go:356] "Generic (PLEG): container finished" podID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerID="fb5a5b68a1b07ccea02be79f30fc98bd6c46a3f41e87f04bf36b9e02bc9a9368" exitCode=0
Sep 16 18:49:56.111005 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.110573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c" event={"ID":"400b28fe-517b-4d2e-ad01-fb5a61ad4576","Type":"ContainerDied","Data":"fb5a5b68a1b07ccea02be79f30fc98bd6c46a3f41e87f04bf36b9e02bc9a9368"}
Sep 16 18:49:56.113182 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.113154    2566 generic.go:356] "Generic (PLEG): container finished" podID="22717e3d-e8b1-4689-a919-a838543238ef" containerID="a6f84dbf3714fab3ac36e8f1babbe5de0c023cca38428b01af40e68934836738" exitCode=0
Sep 16 18:49:56.113302 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.113249    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt" event={"ID":"22717e3d-e8b1-4689-a919-a838543238ef","Type":"ContainerDied","Data":"a6f84dbf3714fab3ac36e8f1babbe5de0c023cca38428b01af40e68934836738"}
Sep 16 18:49:56.116026 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.115866    2566 generic.go:356] "Generic (PLEG): container finished" podID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerID="cc1c7e77d0c99292e0002e1097065b33f862036f514c09f38ea53c603a1ac44f" exitCode=0
Sep 16 18:49:56.116026 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.115930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx" event={"ID":"a846c00b-df9a-4e44-b8be-b76f7146ca48","Type":"ContainerDied","Data":"cc1c7e77d0c99292e0002e1097065b33f862036f514c09f38ea53c603a1ac44f"}
Sep 16 18:49:56.120351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.120325    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bdc6bbd-p7q24"
Sep 16 18:49:56.231614 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.231573    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648469b4d6-mb9jb"]
Sep 16 18:49:56.264371 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.264330    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:56.339912 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.339864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-bundle\") pod \"86d880d4-060f-4ab6-949a-5841d818bbb0\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") "
Sep 16 18:49:56.340044 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.339933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-util\") pod \"86d880d4-060f-4ab6-949a-5841d818bbb0\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") "
Sep 16 18:49:56.340044 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.340001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6r9m\" (UniqueName: \"kubernetes.io/projected/86d880d4-060f-4ab6-949a-5841d818bbb0-kube-api-access-d6r9m\") pod \"86d880d4-060f-4ab6-949a-5841d818bbb0\" (UID: \"86d880d4-060f-4ab6-949a-5841d818bbb0\") "
Sep 16 18:49:56.340644 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.340607    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-bundle" (OuterVolumeSpecName: "bundle") pod "86d880d4-060f-4ab6-949a-5841d818bbb0" (UID: "86d880d4-060f-4ab6-949a-5841d818bbb0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:56.342177 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.342145    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d880d4-060f-4ab6-949a-5841d818bbb0-kube-api-access-d6r9m" (OuterVolumeSpecName: "kube-api-access-d6r9m") pod "86d880d4-060f-4ab6-949a-5841d818bbb0" (UID: "86d880d4-060f-4ab6-949a-5841d818bbb0"). InnerVolumeSpecName "kube-api-access-d6r9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:56.346545 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.346464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-util" (OuterVolumeSpecName: "util") pod "86d880d4-060f-4ab6-949a-5841d818bbb0" (UID: "86d880d4-060f-4ab6-949a-5841d818bbb0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:56.350626 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.350604    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.441432 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.441409    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:56.441529 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.441447    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6r9m\" (UniqueName: \"kubernetes.io/projected/86d880d4-060f-4ab6-949a-5841d818bbb0-kube-api-access-d6r9m\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:56.441529 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.441459    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86d880d4-060f-4ab6-949a-5841d818bbb0-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:56.451249 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.451197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:56.527458 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:56.527425    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:49:57.122218 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.122188    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz"
Sep 16 18:49:57.122601 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.122227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847alf9sz" event={"ID":"86d880d4-060f-4ab6-949a-5841d818bbb0","Type":"ContainerDied","Data":"5ea3551c6dd7f6d71029d7bfc77d37e170627e542469282b07dd9bed986ff6d9"}
Sep 16 18:49:57.122601 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.122277    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea3551c6dd7f6d71029d7bfc77d37e170627e542469282b07dd9bed986ff6d9"
Sep 16 18:49:57.124532 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.124505    2566 generic.go:356] "Generic (PLEG): container finished" podID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerID="d7ea4b3b80d9c9148fe3caa95cb69d95c6d73b4241ad5cdd608b80a36bbf3e0e" exitCode=0
Sep 16 18:49:57.124651 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.124595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c" event={"ID":"400b28fe-517b-4d2e-ad01-fb5a61ad4576","Type":"ContainerDied","Data":"d7ea4b3b80d9c9148fe3caa95cb69d95c6d73b4241ad5cdd608b80a36bbf3e0e"}
Sep 16 18:49:57.264243 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.264219    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:57.284469 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.284442    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:57.350867 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.350838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9wxv\" (UniqueName: \"kubernetes.io/projected/a846c00b-df9a-4e44-b8be-b76f7146ca48-kube-api-access-b9wxv\") pod \"a846c00b-df9a-4e44-b8be-b76f7146ca48\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") "
Sep 16 18:49:57.351014 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.350933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-util\") pod \"a846c00b-df9a-4e44-b8be-b76f7146ca48\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") "
Sep 16 18:49:57.351014 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.350972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-bundle\") pod \"a846c00b-df9a-4e44-b8be-b76f7146ca48\" (UID: \"a846c00b-df9a-4e44-b8be-b76f7146ca48\") "
Sep 16 18:49:57.351501 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.351469    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-bundle" (OuterVolumeSpecName: "bundle") pod "a846c00b-df9a-4e44-b8be-b76f7146ca48" (UID: "a846c00b-df9a-4e44-b8be-b76f7146ca48"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:57.352925 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.352846    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a846c00b-df9a-4e44-b8be-b76f7146ca48-kube-api-access-b9wxv" (OuterVolumeSpecName: "kube-api-access-b9wxv") pod "a846c00b-df9a-4e44-b8be-b76f7146ca48" (UID: "a846c00b-df9a-4e44-b8be-b76f7146ca48"). InnerVolumeSpecName "kube-api-access-b9wxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:57.356565 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.356543    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-util" (OuterVolumeSpecName: "util") pod "a846c00b-df9a-4e44-b8be-b76f7146ca48" (UID: "a846c00b-df9a-4e44-b8be-b76f7146ca48"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:57.451492 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.451470    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-util\") pod \"22717e3d-e8b1-4689-a919-a838543238ef\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") "
Sep 16 18:49:57.451586 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.451505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s5xp\" (UniqueName: \"kubernetes.io/projected/22717e3d-e8b1-4689-a919-a838543238ef-kube-api-access-9s5xp\") pod \"22717e3d-e8b1-4689-a919-a838543238ef\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") "
Sep 16 18:49:57.451586 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.451568    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-bundle\") pod \"22717e3d-e8b1-4689-a919-a838543238ef\" (UID: \"22717e3d-e8b1-4689-a919-a838543238ef\") "
Sep 16 18:49:57.451693 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.451680    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9wxv\" (UniqueName: \"kubernetes.io/projected/a846c00b-df9a-4e44-b8be-b76f7146ca48-kube-api-access-b9wxv\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:57.451729 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.451697    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:57.451729 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.451709    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a846c00b-df9a-4e44-b8be-b76f7146ca48-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:57.452059 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.452039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-bundle" (OuterVolumeSpecName: "bundle") pod "22717e3d-e8b1-4689-a919-a838543238ef" (UID: "22717e3d-e8b1-4689-a919-a838543238ef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:57.453544 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.453517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22717e3d-e8b1-4689-a919-a838543238ef-kube-api-access-9s5xp" (OuterVolumeSpecName: "kube-api-access-9s5xp") pod "22717e3d-e8b1-4689-a919-a838543238ef" (UID: "22717e3d-e8b1-4689-a919-a838543238ef"). InnerVolumeSpecName "kube-api-access-9s5xp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:57.456845 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.456811    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-util" (OuterVolumeSpecName: "util") pod "22717e3d-e8b1-4689-a919-a838543238ef" (UID: "22717e3d-e8b1-4689-a919-a838543238ef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:57.552299 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.552265    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:57.552299 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.552291    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/22717e3d-e8b1-4689-a919-a838543238ef-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:57.552299 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:57.552300    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s5xp\" (UniqueName: \"kubernetes.io/projected/22717e3d-e8b1-4689-a919-a838543238ef-kube-api-access-9s5xp\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:58.131764 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.131732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt" event={"ID":"22717e3d-e8b1-4689-a919-a838543238ef","Type":"ContainerDied","Data":"42f8724ba9c2a08158eebb0ffc03b6e0bed6a8b25f1ac5f6979e649ecce2b69e"}
Sep 16 18:49:58.132244 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.131780    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f8724ba9c2a08158eebb0ffc03b6e0bed6a8b25f1ac5f6979e649ecce2b69e"
Sep 16 18:49:58.132244 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.131747    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c62jcqt"
Sep 16 18:49:58.133705 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.133681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx" event={"ID":"a846c00b-df9a-4e44-b8be-b76f7146ca48","Type":"ContainerDied","Data":"1d3dcaf1d0496364a5eea4ffc66339682605407b3d4b99bb93964212fd9b49e1"}
Sep 16 18:49:58.133705 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.133702    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d76vnxx"
Sep 16 18:49:58.133918 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.133719    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d3dcaf1d0496364a5eea4ffc66339682605407b3d4b99bb93964212fd9b49e1"
Sep 16 18:49:58.267896 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.267852    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:58.359750 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.359716    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-bundle\") pod \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") "
Sep 16 18:49:58.359951 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.359816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpgzj\" (UniqueName: \"kubernetes.io/projected/400b28fe-517b-4d2e-ad01-fb5a61ad4576-kube-api-access-mpgzj\") pod \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") "
Sep 16 18:49:58.359951 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.359846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-util\") pod \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\" (UID: \"400b28fe-517b-4d2e-ad01-fb5a61ad4576\") "
Sep 16 18:49:58.360378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.360349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-bundle" (OuterVolumeSpecName: "bundle") pod "400b28fe-517b-4d2e-ad01-fb5a61ad4576" (UID: "400b28fe-517b-4d2e-ad01-fb5a61ad4576"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:58.361949 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.361921    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400b28fe-517b-4d2e-ad01-fb5a61ad4576-kube-api-access-mpgzj" (OuterVolumeSpecName: "kube-api-access-mpgzj") pod "400b28fe-517b-4d2e-ad01-fb5a61ad4576" (UID: "400b28fe-517b-4d2e-ad01-fb5a61ad4576"). InnerVolumeSpecName "kube-api-access-mpgzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:49:58.432686 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.432636    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-util" (OuterVolumeSpecName: "util") pod "400b28fe-517b-4d2e-ad01-fb5a61ad4576" (UID: "400b28fe-517b-4d2e-ad01-fb5a61ad4576"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:49:58.461372 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.461347    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpgzj\" (UniqueName: \"kubernetes.io/projected/400b28fe-517b-4d2e-ad01-fb5a61ad4576-kube-api-access-mpgzj\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:58.461372 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.461371    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-util\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:58.461502 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:58.461380    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/400b28fe-517b-4d2e-ad01-fb5a61ad4576-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:49:59.140669 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:59.140641    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c"
Sep 16 18:49:59.140669 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:59.140652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdl28c" event={"ID":"400b28fe-517b-4d2e-ad01-fb5a61ad4576","Type":"ContainerDied","Data":"72eec2b7b39f644332d085c0aa0f57ccff4654c43d5d489a32841c15a3d63da7"}
Sep 16 18:49:59.141118 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:49:59.140695    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72eec2b7b39f644332d085c0aa0f57ccff4654c43d5d489a32841c15a3d63da7"
Sep 16 18:50:04.737559 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:04.737519    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:06.826497 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.826458    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"]
Sep 16 18:50:06.827528 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827496    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="util"
Sep 16 18:50:06.827528 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827524    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="util"
Sep 16 18:50:06.827528 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827537    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827550    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827555    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827562    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="util"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827567    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="util"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827575    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="util"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827580    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="util"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827585    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827590    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827596    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827601    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827607    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827613    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="pull"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827620    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827626    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827632    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827636    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827643    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827647    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827660    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="util"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827669    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="util"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827743    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="400b28fe-517b-4d2e-ad01-fb5a61ad4576" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827755    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="22717e3d-e8b1-4689-a919-a838543238ef" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827766    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86d880d4-060f-4ab6-949a-5841d818bbb0" containerName="extract"
Sep 16 18:50:06.827792 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.827777    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a846c00b-df9a-4e44-b8be-b76f7146ca48" containerName="extract"
Sep 16 18:50:06.836274 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.836252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:06.839699 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.839674    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 16 18:50:06.839928 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.839903    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qf4fg\""
Sep 16 18:50:06.840064 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.840043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"]
Sep 16 18:50:06.840169 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.840153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 16 18:50:06.930443 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:06.930410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4csq\" (UniqueName: \"kubernetes.io/projected/97e01e16-c64f-4213-88e3-158692641640-kube-api-access-b4csq\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8tx67\" (UID: \"97e01e16-c64f-4213-88e3-158692641640\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:07.031814 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:07.031775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4csq\" (UniqueName: \"kubernetes.io/projected/97e01e16-c64f-4213-88e3-158692641640-kube-api-access-b4csq\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8tx67\" (UID: \"97e01e16-c64f-4213-88e3-158692641640\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:07.052588 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:07.052561    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4csq\" (UniqueName: \"kubernetes.io/projected/97e01e16-c64f-4213-88e3-158692641640-kube-api-access-b4csq\") pod \"limitador-operator-controller-manager-5ffb4b4ccb-8tx67\" (UID: \"97e01e16-c64f-4213-88e3-158692641640\") " pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:07.147289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:07.147257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:07.429239 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:07.429164    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"]
Sep 16 18:50:07.432531 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:50:07.432505    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e01e16_c64f_4213_88e3_158692641640.slice/crio-eedb64f1513e38d5fb02a9a22f30b5f32cc73b7e802014d3a4f3107384691307 WatchSource:0}: Error finding container eedb64f1513e38d5fb02a9a22f30b5f32cc73b7e802014d3a4f3107384691307: Status 404 returned error can't find the container with id eedb64f1513e38d5fb02a9a22f30b5f32cc73b7e802014d3a4f3107384691307
Sep 16 18:50:07.557865 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:07.557828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:07.943305 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:07.943273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:08.183955 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:08.183921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67" event={"ID":"97e01e16-c64f-4213-88e3-158692641640","Type":"ContainerStarted","Data":"eedb64f1513e38d5fb02a9a22f30b5f32cc73b7e802014d3a4f3107384691307"}
Sep 16 18:50:08.340378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:08.340295    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:11.098698 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.098664    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"]
Sep 16 18:50:11.103091 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.103072    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.107252 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.107227    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9wbnh\""
Sep 16 18:50:11.120445 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.120418    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"]
Sep 16 18:50:11.262074 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.262045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzcn\" (UniqueName: \"kubernetes.io/projected/9ff329cd-9941-49e0-aaee-8f2467221ed6-kube-api-access-rjzcn\") pod \"kuadrant-operator-controller-manager-6d4cb79c-f4lrw\" (UID: \"9ff329cd-9941-49e0-aaee-8f2467221ed6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.262223 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.262113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ff329cd-9941-49e0-aaee-8f2467221ed6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-f4lrw\" (UID: \"9ff329cd-9941-49e0-aaee-8f2467221ed6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.363389 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.363317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzcn\" (UniqueName: \"kubernetes.io/projected/9ff329cd-9941-49e0-aaee-8f2467221ed6-kube-api-access-rjzcn\") pod \"kuadrant-operator-controller-manager-6d4cb79c-f4lrw\" (UID: \"9ff329cd-9941-49e0-aaee-8f2467221ed6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.363389 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.363371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ff329cd-9941-49e0-aaee-8f2467221ed6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-f4lrw\" (UID: \"9ff329cd-9941-49e0-aaee-8f2467221ed6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.363701 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.363684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9ff329cd-9941-49e0-aaee-8f2467221ed6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6d4cb79c-f4lrw\" (UID: \"9ff329cd-9941-49e0-aaee-8f2467221ed6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.374171 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.374147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzcn\" (UniqueName: \"kubernetes.io/projected/9ff329cd-9941-49e0-aaee-8f2467221ed6-kube-api-access-rjzcn\") pod \"kuadrant-operator-controller-manager-6d4cb79c-f4lrw\" (UID: \"9ff329cd-9941-49e0-aaee-8f2467221ed6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.381143 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.381110    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:11.413455 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.413422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:11.607575 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.607535    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:11.715207 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.711533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"]
Sep 16 18:50:11.718469 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:50:11.718432    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ff329cd_9941_49e0_aaee_8f2467221ed6.slice/crio-1b22d1307cbff597a1b74d1b4331542f5fb8dba3c4ee6ab6cad45a02af183096 WatchSource:0}: Error finding container 1b22d1307cbff597a1b74d1b4331542f5fb8dba3c4ee6ab6cad45a02af183096: Status 404 returned error can't find the container with id 1b22d1307cbff597a1b74d1b4331542f5fb8dba3c4ee6ab6cad45a02af183096
Sep 16 18:50:11.751808 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:11.751779    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:12.205967 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:12.205928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw" event={"ID":"9ff329cd-9941-49e0-aaee-8f2467221ed6","Type":"ContainerStarted","Data":"1b22d1307cbff597a1b74d1b4331542f5fb8dba3c4ee6ab6cad45a02af183096"}
Sep 16 18:50:12.638077 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:12.638040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:12.776655 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:12.776621    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:13.216074 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:13.216028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67" event={"ID":"97e01e16-c64f-4213-88e3-158692641640","Type":"ContainerStarted","Data":"c659047134e7f2d47ae57f40577e2ff47489c19b6755c70e9958a0917b8fc87d"}
Sep 16 18:50:13.216543 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:13.216165    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:13.297754 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:13.297687    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67" podStartSLOduration=2.079142926 podStartE2EDuration="7.297666686s" totalImagesPullingTime="5.21852376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:13.297584012 +0000 UTC m=+397.409380655" watchObservedRunningTime="2026-09-16 18:50:13.297666686 +0000 UTC m=+397.409463328"
Sep 16 18:50:14.299321 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:14.299276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:14.561076 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:14.560988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:15.655022 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:15.654997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:16.235912 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:16.235867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw" event={"ID":"9ff329cd-9941-49e0-aaee-8f2467221ed6","Type":"ContainerStarted","Data":"b737b84004f7ff524bd9a06dba3281fcefdac66b55bf735475650bd2e4b6bc59"}
Sep 16 18:50:16.236130 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:16.236036    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:16.266254 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:16.266207    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw" podStartSLOduration=1.356859024 podStartE2EDuration="5.266191145s" totalImagesPullingTime="3.909332121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:16.265040695 +0000 UTC m=+400.376837338" watchObservedRunningTime="2026-09-16 18:50:16.266191145 +0000 UTC m=+400.377987786"
Sep 16 18:50:17.552369 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:17.552335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:17.648666 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:17.648621    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:17.978125 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:17.978093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:18.325824 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:18.325734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:21.253664 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.253614    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-648469b4d6-mb9jb" podUID="e67eb7c8-649e-4d76-a7af-9176b7997839" containerName="console" containerID="cri-o://af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4" gracePeriod=15
Sep 16 18:50:21.497967 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.497942    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:50:21.545934 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.545847    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-trusted-ca-bundle\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.545934 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.545894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-serving-cert\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.545934 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.545919    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-console-config\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.546150 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.545990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-oauth-serving-cert\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.546150 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-service-ca\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.546150 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546083    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttw9x\" (UniqueName: \"kubernetes.io/projected/e67eb7c8-649e-4d76-a7af-9176b7997839-kube-api-access-ttw9x\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.546150 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-oauth-config\") pod \"e67eb7c8-649e-4d76-a7af-9176b7997839\" (UID: \"e67eb7c8-649e-4d76-a7af-9176b7997839\") "
Sep 16 18:50:21.546378 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:21.546516 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546490    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:21.546582 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-service-ca" (OuterVolumeSpecName: "service-ca") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:21.546636 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546604    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-trusted-ca-bundle\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.546636 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546625    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-oauth-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.546717 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546642    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-service-ca\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.546717 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.546668    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-console-config" (OuterVolumeSpecName: "console-config") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:50:21.548274 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.548243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67eb7c8-649e-4d76-a7af-9176b7997839-kube-api-access-ttw9x" (OuterVolumeSpecName: "kube-api-access-ttw9x") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "kube-api-access-ttw9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:50:21.548502 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.548479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:21.548570 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.548500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e67eb7c8-649e-4d76-a7af-9176b7997839" (UID: "e67eb7c8-649e-4d76-a7af-9176b7997839"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:50:21.647978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.647942    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttw9x\" (UniqueName: \"kubernetes.io/projected/e67eb7c8-649e-4d76-a7af-9176b7997839-kube-api-access-ttw9x\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.647978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.647966    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-oauth-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.647978 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.647980    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e67eb7c8-649e-4d76-a7af-9176b7997839-console-serving-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.648174 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.647992    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e67eb7c8-649e-4d76-a7af-9176b7997839-console-config\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:50:21.737470 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.737440    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:21.839068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:21.838984    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:22.253115 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.253084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:22.273717 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.273685    2566 generic.go:356] "Generic (PLEG): container finished" podID="e67eb7c8-649e-4d76-a7af-9176b7997839" containerID="af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4" exitCode=0
Sep 16 18:50:22.274141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.273727    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-648469b4d6-mb9jb"
Sep 16 18:50:22.274141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.273733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648469b4d6-mb9jb" event={"ID":"e67eb7c8-649e-4d76-a7af-9176b7997839","Type":"ContainerDied","Data":"af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4"}
Sep 16 18:50:22.274141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.273771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-648469b4d6-mb9jb" event={"ID":"e67eb7c8-649e-4d76-a7af-9176b7997839","Type":"ContainerDied","Data":"d1f866b1693c62835c88b95333e77ef651de3437918975608e3175be38e02820"}
Sep 16 18:50:22.274141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.273791    2566 scope.go:117] "RemoveContainer" containerID="af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4"
Sep 16 18:50:22.283447 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.283423    2566 scope.go:117] "RemoveContainer" containerID="af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4"
Sep 16 18:50:22.283739 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:50:22.283719    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4\": container with ID starting with af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4 not found: ID does not exist" containerID="af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4"
Sep 16 18:50:22.283795 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.283747    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4"} err="failed to get container status \"af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4\": rpc error: code = NotFound desc = could not find container \"af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4\": container with ID starting with af64936daa7cb65507ad97131a237a535bbd73becb5ecd9cb4272f68161cb3a4 not found: ID does not exist"
Sep 16 18:50:22.322171 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.322127    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-648469b4d6-mb9jb"]
Sep 16 18:50:22.335219 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.335190    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-648469b4d6-mb9jb"]
Sep 16 18:50:22.397914 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.397844    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:22.402493 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.402468    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e67eb7c8-649e-4d76-a7af-9176b7997839" path="/var/lib/kubelet/pods/e67eb7c8-649e-4d76-a7af-9176b7997839/volumes"
Sep 16 18:50:22.503254 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:22.503166    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:24.224936 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:24.224900    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5ffb4b4ccb-8tx67"
Sep 16 18:50:27.244531 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:27.244500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6d4cb79c-f4lrw"
Sep 16 18:50:28.974340 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:28.974302    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:29.074750 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:29.074707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:30.252504 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:30.252466    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:30.362356 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:30.362321    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:32.652269 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:32.652235    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:32.711684 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:32.711625    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:37.052445 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:37.052386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:37.156081 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:37.156045    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:40.026062 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:40.026020    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:40.617816 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:40.617774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:40.686081 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:40.686041    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:41.020928 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:41.020865    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:41.121128 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:41.121096    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:41.223415 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:41.223363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:41.396351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:41.396308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:50.039779 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:50.039744    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:50.166136 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:50.166104    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:53.609417 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:53.609375    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:53.682113 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:53.682084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.396637 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.396541    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.409384 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.409349    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:50:59.409674 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.409661    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e67eb7c8-649e-4d76-a7af-9176b7997839" containerName="console"
Sep 16 18:50:59.409674 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.409675    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e67eb7c8-649e-4d76-a7af-9176b7997839" containerName="console"
Sep 16 18:50:59.409775 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.409737    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e67eb7c8-649e-4d76-a7af-9176b7997839" containerName="console"
Sep 16 18:50:59.411906 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.411860    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.417986 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.417957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 16 18:50:59.418119 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.418012    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zj67w\""
Sep 16 18:50:59.487828 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.487794    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:50:59.516441 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.516408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:50:59.528509 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.528473    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:50:59.537463 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.537431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5g45\" (UniqueName: \"kubernetes.io/projected/56815809-1302-423c-9e70-5f615c3163b1-kube-api-access-d5g45\") pod \"limitador-limitador-8588749977-xpzfh\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") " pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.537596 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.537486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56815809-1302-423c-9e70-5f615c3163b1-config-file\") pod \"limitador-limitador-8588749977-xpzfh\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") " pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.638632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.638595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5g45\" (UniqueName: \"kubernetes.io/projected/56815809-1302-423c-9e70-5f615c3163b1-kube-api-access-d5g45\") pod \"limitador-limitador-8588749977-xpzfh\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") " pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.638821 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.638642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56815809-1302-423c-9e70-5f615c3163b1-config-file\") pod \"limitador-limitador-8588749977-xpzfh\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") " pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.639261 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.639241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56815809-1302-423c-9e70-5f615c3163b1-config-file\") pod \"limitador-limitador-8588749977-xpzfh\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") " pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.649828 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.649756    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5g45\" (UniqueName: \"kubernetes.io/projected/56815809-1302-423c-9e70-5f615c3163b1-kube-api-access-d5g45\") pod \"limitador-limitador-8588749977-xpzfh\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") " pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.722537 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.722506    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:50:59.998741 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:50:59.998716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:51:00.000751 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:51:00.000713    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56815809_1302_423c_9e70_5f615c3163b1.slice/crio-c85390a802a9d2da806d949cf72a42596fbcdb333a7e1724380010cc35923f11 WatchSource:0}: Error finding container c85390a802a9d2da806d949cf72a42596fbcdb333a7e1724380010cc35923f11: Status 404 returned error can't find the container with id c85390a802a9d2da806d949cf72a42596fbcdb333a7e1724380010cc35923f11
Sep 16 18:51:00.382260 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.382168    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:00.457722 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.457686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh" event={"ID":"56815809-1302-423c-9e70-5f615c3163b1","Type":"ContainerStarted","Data":"c85390a802a9d2da806d949cf72a42596fbcdb333a7e1724380010cc35923f11"}
Sep 16 18:51:00.491136 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.491109    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:00.755287 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.755255    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-jmml4"]
Sep 16 18:51:00.758304 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.758287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:00.761914 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.761871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l4psz\""
Sep 16 18:51:00.776197 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.776174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jmml4"]
Sep 16 18:51:00.848778 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.848755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvghn\" (UniqueName: \"kubernetes.io/projected/1dfafdb2-f2df-46fd-b0ff-6877da422550-kube-api-access-mvghn\") pod \"authorino-59c974bd48-jmml4\" (UID: \"1dfafdb2-f2df-46fd-b0ff-6877da422550\") " pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:00.949747 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.949721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvghn\" (UniqueName: \"kubernetes.io/projected/1dfafdb2-f2df-46fd-b0ff-6877da422550-kube-api-access-mvghn\") pod \"authorino-59c974bd48-jmml4\" (UID: \"1dfafdb2-f2df-46fd-b0ff-6877da422550\") " pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:00.959031 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:00.959005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvghn\" (UniqueName: \"kubernetes.io/projected/1dfafdb2-f2df-46fd-b0ff-6877da422550-kube-api-access-mvghn\") pod \"authorino-59c974bd48-jmml4\" (UID: \"1dfafdb2-f2df-46fd-b0ff-6877da422550\") " pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:01.067633 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:01.067567    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:01.344802 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:01.344773    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jmml4"]
Sep 16 18:51:01.494777 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:51:01.494743    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dfafdb2_f2df_46fd_b0ff_6877da422550.slice/crio-406943aca8ce2a1639371df17f2fe5a1c1ba42985639d07b32042462cab8fda4 WatchSource:0}: Error finding container 406943aca8ce2a1639371df17f2fe5a1c1ba42985639d07b32042462cab8fda4: Status 404 returned error can't find the container with id 406943aca8ce2a1639371df17f2fe5a1c1ba42985639d07b32042462cab8fda4
Sep 16 18:51:02.470557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:02.470520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh" event={"ID":"56815809-1302-423c-9e70-5f615c3163b1","Type":"ContainerStarted","Data":"bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc"}
Sep 16 18:51:02.470752 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:02.470626    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:51:02.471912 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:02.471868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jmml4" event={"ID":"1dfafdb2-f2df-46fd-b0ff-6877da422550","Type":"ContainerStarted","Data":"406943aca8ce2a1639371df17f2fe5a1c1ba42985639d07b32042462cab8fda4"}
Sep 16 18:51:02.489112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:02.489070    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh" podStartSLOduration=1.9630411159999999 podStartE2EDuration="3.489053676s" totalImagesPullingTime="1.52601256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:02.487023112 +0000 UTC m=+446.598819754" watchObservedRunningTime="2026-09-16 18:51:02.489053676 +0000 UTC m=+446.600850514"
Sep 16 18:51:04.483226 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:04.483188    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jmml4" event={"ID":"1dfafdb2-f2df-46fd-b0ff-6877da422550","Type":"ContainerStarted","Data":"24a64429fcbb836b6e4b80996e970e4da019f929ac20f4f4a96830fa9f73bef4"}
Sep 16 18:51:04.521793 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:04.521736    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-jmml4" podStartSLOduration=2.103668615 podStartE2EDuration="4.521722423s" totalImagesPullingTime="2.418053808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:04.520779279 +0000 UTC m=+448.632575921" watchObservedRunningTime="2026-09-16 18:51:04.521722423 +0000 UTC m=+448.633519064"
Sep 16 18:51:13.478705 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:13.478670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:51:13.894296 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:13.894251    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:51:13.894558 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:13.894511    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh" podUID="56815809-1302-423c-9e70-5f615c3163b1" containerName="limitador" containerID="cri-o://bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc" gracePeriod=30
Sep 16 18:51:14.434289 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.434263    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:51:14.534854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.534779    2566 generic.go:356] "Generic (PLEG): container finished" podID="56815809-1302-423c-9e70-5f615c3163b1" containerID="bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc" exitCode=0
Sep 16 18:51:14.534854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.534816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh" event={"ID":"56815809-1302-423c-9e70-5f615c3163b1","Type":"ContainerDied","Data":"bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc"}
Sep 16 18:51:14.534854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.534822    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh"
Sep 16 18:51:14.534854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.534851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-xpzfh" event={"ID":"56815809-1302-423c-9e70-5f615c3163b1","Type":"ContainerDied","Data":"c85390a802a9d2da806d949cf72a42596fbcdb333a7e1724380010cc35923f11"}
Sep 16 18:51:14.535367 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.534874    2566 scope.go:117] "RemoveContainer" containerID="bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc"
Sep 16 18:51:14.543334 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.543317    2566 scope.go:117] "RemoveContainer" containerID="bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc"
Sep 16 18:51:14.543561 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:51:14.543540    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc\": container with ID starting with bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc not found: ID does not exist" containerID="bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc"
Sep 16 18:51:14.543608 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.543569    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc"} err="failed to get container status \"bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc\": rpc error: code = NotFound desc = could not find container \"bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc\": container with ID starting with bdaf2c7d359016529836a54571adabff42038bc1613c84ccedf73bd9c3d120dc not found: ID does not exist"
Sep 16 18:51:14.554953 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.554933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d5g45\" (UniqueName: \"kubernetes.io/projected/56815809-1302-423c-9e70-5f615c3163b1-kube-api-access-d5g45\") pod \"56815809-1302-423c-9e70-5f615c3163b1\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") "
Sep 16 18:51:14.555036 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.554978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56815809-1302-423c-9e70-5f615c3163b1-config-file\") pod \"56815809-1302-423c-9e70-5f615c3163b1\" (UID: \"56815809-1302-423c-9e70-5f615c3163b1\") "
Sep 16 18:51:14.555307 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.555289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56815809-1302-423c-9e70-5f615c3163b1-config-file" (OuterVolumeSpecName: "config-file") pod "56815809-1302-423c-9e70-5f615c3163b1" (UID: "56815809-1302-423c-9e70-5f615c3163b1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:14.557049 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.557027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56815809-1302-423c-9e70-5f615c3163b1-kube-api-access-d5g45" (OuterVolumeSpecName: "kube-api-access-d5g45") pod "56815809-1302-423c-9e70-5f615c3163b1" (UID: "56815809-1302-423c-9e70-5f615c3163b1"). InnerVolumeSpecName "kube-api-access-d5g45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:14.656325 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.656291    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5g45\" (UniqueName: \"kubernetes.io/projected/56815809-1302-423c-9e70-5f615c3163b1-kube-api-access-d5g45\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:14.656325 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.656318    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56815809-1302-423c-9e70-5f615c3163b1-config-file\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:14.857489 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.857452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:51:14.861261 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:14.861236    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-xpzfh"]
Sep 16 18:51:16.402028 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:16.401989    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56815809-1302-423c-9e70-5f615c3163b1" path="/var/lib/kubelet/pods/56815809-1302-423c-9e70-5f615c3163b1/volumes"
Sep 16 18:51:19.987443 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:19.987409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:20.250666 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:20.250582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:25.425578 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.425548    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jmml4"]
Sep 16 18:51:25.426003 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.425716    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-jmml4" podUID="1dfafdb2-f2df-46fd-b0ff-6877da422550" containerName="authorino" containerID="cri-o://24a64429fcbb836b6e4b80996e970e4da019f929ac20f4f4a96830fa9f73bef4" gracePeriod=30
Sep 16 18:51:25.589189 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.589157    2566 generic.go:356] "Generic (PLEG): container finished" podID="1dfafdb2-f2df-46fd-b0ff-6877da422550" containerID="24a64429fcbb836b6e4b80996e970e4da019f929ac20f4f4a96830fa9f73bef4" exitCode=0
Sep 16 18:51:25.589385 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.589215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jmml4" event={"ID":"1dfafdb2-f2df-46fd-b0ff-6877da422550","Type":"ContainerDied","Data":"24a64429fcbb836b6e4b80996e970e4da019f929ac20f4f4a96830fa9f73bef4"}
Sep 16 18:51:25.688016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.687960    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:25.844460 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.844431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mvghn\" (UniqueName: \"kubernetes.io/projected/1dfafdb2-f2df-46fd-b0ff-6877da422550-kube-api-access-mvghn\") pod \"1dfafdb2-f2df-46fd-b0ff-6877da422550\" (UID: \"1dfafdb2-f2df-46fd-b0ff-6877da422550\") "
Sep 16 18:51:25.846409 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.846380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfafdb2-f2df-46fd-b0ff-6877da422550-kube-api-access-mvghn" (OuterVolumeSpecName: "kube-api-access-mvghn") pod "1dfafdb2-f2df-46fd-b0ff-6877da422550" (UID: "1dfafdb2-f2df-46fd-b0ff-6877da422550"). InnerVolumeSpecName "kube-api-access-mvghn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:25.945027 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:25.944968    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvghn\" (UniqueName: \"kubernetes.io/projected/1dfafdb2-f2df-46fd-b0ff-6877da422550-kube-api-access-mvghn\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:26.595416 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:26.595387    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-jmml4"
Sep 16 18:51:26.595809 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:26.595387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-jmml4" event={"ID":"1dfafdb2-f2df-46fd-b0ff-6877da422550","Type":"ContainerDied","Data":"406943aca8ce2a1639371df17f2fe5a1c1ba42985639d07b32042462cab8fda4"}
Sep 16 18:51:26.595809 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:26.595508    2566 scope.go:117] "RemoveContainer" containerID="24a64429fcbb836b6e4b80996e970e4da019f929ac20f4f4a96830fa9f73bef4"
Sep 16 18:51:26.614591 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:26.614559    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jmml4"]
Sep 16 18:51:26.618893 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:26.618860    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-jmml4"]
Sep 16 18:51:28.402574 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:28.402530    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dfafdb2-f2df-46fd-b0ff-6877da422550" path="/var/lib/kubelet/pods/1dfafdb2-f2df-46fd-b0ff-6877da422550/volumes"
Sep 16 18:51:29.023405 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:29.023372    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:29.109207 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:29.109177    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:45.017108 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017073    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"]
Sep 16 18:51:45.017557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017404    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56815809-1302-423c-9e70-5f615c3163b1" containerName="limitador"
Sep 16 18:51:45.017557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017417    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="56815809-1302-423c-9e70-5f615c3163b1" containerName="limitador"
Sep 16 18:51:45.017557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017433    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dfafdb2-f2df-46fd-b0ff-6877da422550" containerName="authorino"
Sep 16 18:51:45.017557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017441    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dfafdb2-f2df-46fd-b0ff-6877da422550" containerName="authorino"
Sep 16 18:51:45.017557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017495    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="56815809-1302-423c-9e70-5f615c3163b1" containerName="limitador"
Sep 16 18:51:45.017557 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.017504    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dfafdb2-f2df-46fd-b0ff-6877da422550" containerName="authorino"
Sep 16 18:51:45.020358 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.020340    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.037975 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.037942    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"]
Sep 16 18:51:45.097864 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.097831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rcc\" (UniqueName: \"kubernetes.io/projected/78a6b5a3-3645-4693-a957-3354e0c592ca-kube-api-access-p6rcc\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.098047 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.097870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a6b5a3-3645-4693-a957-3354e0c592ca-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.098047 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.097955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.098047 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.097974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.098047 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.097992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.098047 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.098014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.098221 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.098091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.198839 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.198799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.199039 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.198904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rcc\" (UniqueName: \"kubernetes.io/projected/78a6b5a3-3645-4693-a957-3354e0c592ca-kube-api-access-p6rcc\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.199083 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.199048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a6b5a3-3645-4693-a957-3354e0c592ca-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.199120 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.199106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.199162 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.199126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.199162 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.199147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.199266 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.199172    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.200034 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.200002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.201534 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.201498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78a6b5a3-3645-4693-a957-3354e0c592ca-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.201534 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.201518    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.201686 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.201645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.201725 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.201680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.207276 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.207250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78a6b5a3-3645-4693-a957-3354e0c592ca-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.208266 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.208251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rcc\" (UniqueName: \"kubernetes.io/projected/78a6b5a3-3645-4693-a957-3354e0c592ca-kube-api-access-p6rcc\") pod \"istiod-openshift-gateway-5677ccb49b-mkxxm\" (UID: \"78a6b5a3-3645-4693-a957-3354e0c592ca\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.329068 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.328974    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:45.626445 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.626357    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"]
Sep 16 18:51:45.630462 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:51:45.630430    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a6b5a3_3645_4693_a957_3354e0c592ca.slice/crio-6aec4754c6cac694f8ba6fa00d12d4acf0d84be6aa9fd8f1b04a2ed837e8564c WatchSource:0}: Error finding container 6aec4754c6cac694f8ba6fa00d12d4acf0d84be6aa9fd8f1b04a2ed837e8564c: Status 404 returned error can't find the container with id 6aec4754c6cac694f8ba6fa00d12d4acf0d84be6aa9fd8f1b04a2ed837e8564c
Sep 16 18:51:45.632652 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.632606    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:51:45.632758 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.632691    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 18:51:45.690381 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:45.690351    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm" event={"ID":"78a6b5a3-3645-4693-a957-3354e0c592ca","Type":"ContainerStarted","Data":"6aec4754c6cac694f8ba6fa00d12d4acf0d84be6aa9fd8f1b04a2ed837e8564c"}
Sep 16 18:51:46.697193 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:46.697152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm" event={"ID":"78a6b5a3-3645-4693-a957-3354e0c592ca","Type":"ContainerStarted","Data":"12f233f9c1555153b503ec3cfdf855b2d08cb228d0263edece7494fc2d109b4b"}
Sep 16 18:51:46.697687 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:46.697319    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:46.699035 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:46.699008    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-mkxxm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 16 18:51:46.699158 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:46.699065    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm" podUID="78a6b5a3-3645-4693-a957-3354e0c592ca" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:51:46.723894 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:46.723840    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm" podStartSLOduration=2.7238264819999998 podStartE2EDuration="2.723826482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:51:46.7228199 +0000 UTC m=+490.834616541" watchObservedRunningTime="2026-09-16 18:51:46.723826482 +0000 UTC m=+490.835623122"
Sep 16 18:51:47.703143 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:47.703114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 18:51:47.756714 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:47.756682    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"]
Sep 16 18:51:47.756915 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:47.756874    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" podUID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" containerName="discovery" containerID="cri-o://5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223" gracePeriod=30
Sep 16 18:51:47.997562 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:47.997537    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:51:48.125218 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-token\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-local-certs\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-kubeconfig\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-cacerts\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125322    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lblq5\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-kube-api-access-lblq5\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125345    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-dns-cert\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125412 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125387    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-ca-configmap\") pod \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\" (UID: \"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90\") "
Sep 16 18:51:48.125916 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.125866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:51:48.128234 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.128200    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-kube-api-access-lblq5" (OuterVolumeSpecName: "kube-api-access-lblq5") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "kube-api-access-lblq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:48.128342 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.128287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-local-certs" (OuterVolumeSpecName: "local-certs") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:51:48.128342 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.128281    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:48.128342 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.128292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-cacerts" (OuterVolumeSpecName: "cacerts") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:48.128342 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.128291    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-token" (OuterVolumeSpecName: "istio-token") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:51:48.128342 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.128318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" (UID: "4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:51:48.226935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226889    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-ca-configmap\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.226935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226931    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-token\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.226935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226942    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-local-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.226935 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226950    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-kubeconfig\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.227196 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226958    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-cacerts\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.227196 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226968    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lblq5\" (UniqueName: \"kubernetes.io/projected/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-kube-api-access-lblq5\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.227196 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.226977    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90-istio-csr-dns-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:51:48.709016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.708923    2566 generic.go:356] "Generic (PLEG): container finished" podID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" containerID="5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223" exitCode=0
Sep 16 18:51:48.709016 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.708976    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"
Sep 16 18:51:48.709519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.709015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" event={"ID":"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90","Type":"ContainerDied","Data":"5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223"}
Sep 16 18:51:48.709519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.709071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" event={"ID":"4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90","Type":"ContainerDied","Data":"fd601fc23b088dc8e653c9ff76436a72d557f6fa315bf5917a162ba2befb0ec8"}
Sep 16 18:51:48.709519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.709101    2566 scope.go:117] "RemoveContainer" containerID="5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223"
Sep 16 18:51:48.717553 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.717534    2566 scope.go:117] "RemoveContainer" containerID="5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223"
Sep 16 18:51:48.717825 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:51:48.717807    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223\": container with ID starting with 5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223 not found: ID does not exist" containerID="5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223"
Sep 16 18:51:48.717861 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.717835    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223"} err="failed to get container status \"5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223\": rpc error: code = NotFound desc = could not find container \"5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223\": container with ID starting with 5292dbced43294de585df7be6c69830830215d9cc84cee51f096c21ee6a62223 not found: ID does not exist"
Sep 16 18:51:48.750189 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.750160    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"]
Sep 16 18:51:48.768874 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:48.768851    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl"]
Sep 16 18:51:49.831323 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:49.831285    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:49.931085 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:49.931054    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:50.401897 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:50.401847    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" path="/var/lib/kubelet/pods/4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90/volumes"
Sep 16 18:51:52.971072 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:52.971031    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bp8pl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.29:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 16 18:51:52.971505 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:52.971098    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8pl" podUID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.29:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 18:51:58.214310 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.214256    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:58.295171 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.295135    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-7jnv6"]
Sep 16 18:51:58.295638 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.295625    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" containerName="discovery"
Sep 16 18:51:58.295683 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.295642    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" containerName="discovery"
Sep 16 18:51:58.295730 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.295719    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c9e23d7-f682-4d9b-9ca5-4b9650fb1e90" containerName="discovery"
Sep 16 18:51:58.298640 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.298618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.301797 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.301777    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qnk2r\""
Sep 16 18:51:58.302063 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.302047    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 16 18:51:58.303060 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.303042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:51:58.303227 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.303204    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:51:58.316495 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.316457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:58.317429 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.317404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-7jnv6"]
Sep 16 18:51:58.319596 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.319575    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"]
Sep 16 18:51:58.322305 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.322175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:58.324942 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.324922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-l79nx\""
Sep 16 18:51:58.325138 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.325115    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 16 18:51:58.333481 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.333464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"]
Sep 16 18:51:58.406034 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.406002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgm7\" (UniqueName: \"kubernetes.io/projected/a3c72322-36a0-4841-aeb4-e6156d441e1f-kube-api-access-rxgm7\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:58.406204 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.406039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c72322-36a0-4841-aeb4-e6156d441e1f-cert\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:58.406204 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.406079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49606616-60a9-4e85-9e9c-5e2a2b13976d-cert\") pod \"kserve-controller-manager-f8c46b677-7jnv6\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") " pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.406204 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.406100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bc74\" (UniqueName: \"kubernetes.io/projected/49606616-60a9-4e85-9e9c-5e2a2b13976d-kube-api-access-7bc74\") pod \"kserve-controller-manager-f8c46b677-7jnv6\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") " pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.410386 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.410358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:58.507083 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.506990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49606616-60a9-4e85-9e9c-5e2a2b13976d-cert\") pod \"kserve-controller-manager-f8c46b677-7jnv6\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") " pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.507083 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.507043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bc74\" (UniqueName: \"kubernetes.io/projected/49606616-60a9-4e85-9e9c-5e2a2b13976d-kube-api-access-7bc74\") pod \"kserve-controller-manager-f8c46b677-7jnv6\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") " pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.507311 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.507116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgm7\" (UniqueName: \"kubernetes.io/projected/a3c72322-36a0-4841-aeb4-e6156d441e1f-kube-api-access-rxgm7\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:58.507376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.507313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c72322-36a0-4841-aeb4-e6156d441e1f-cert\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:58.507451 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:51:58.507431    2566 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 16 18:51:58.507520 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:51:58.507507    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3c72322-36a0-4841-aeb4-e6156d441e1f-cert podName:a3c72322-36a0-4841-aeb4-e6156d441e1f nodeName:}" failed. No retries permitted until 2026-09-16 18:51:59.007486884 +0000 UTC m=+503.119283505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/a3c72322-36a0-4841-aeb4-e6156d441e1f-cert") pod "llmisvc-controller-manager-c98cf57d5-wmmtx" (UID: "a3c72322-36a0-4841-aeb4-e6156d441e1f") : secret "llmisvc-webhook-server-cert" not found
Sep 16 18:51:58.509509 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.509474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:58.509920 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.509899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49606616-60a9-4e85-9e9c-5e2a2b13976d-cert\") pod \"kserve-controller-manager-f8c46b677-7jnv6\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") " pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.516989 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.516964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bc74\" (UniqueName: \"kubernetes.io/projected/49606616-60a9-4e85-9e9c-5e2a2b13976d-kube-api-access-7bc74\") pod \"kserve-controller-manager-f8c46b677-7jnv6\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") " pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.517367 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.517349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgm7\" (UniqueName: \"kubernetes.io/projected/a3c72322-36a0-4841-aeb4-e6156d441e1f-kube-api-access-rxgm7\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:58.609085 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.609056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:51:58.669197 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.669098    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:58.894973 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:51:58.894945    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49606616_60a9_4e85_9e9c_5e2a2b13976d.slice/crio-b1f7e2e8c176060026f88103913afee46d9db6514188aac2f0ee942e16fdef6c WatchSource:0}: Error finding container b1f7e2e8c176060026f88103913afee46d9db6514188aac2f0ee942e16fdef6c: Status 404 returned error can't find the container with id b1f7e2e8c176060026f88103913afee46d9db6514188aac2f0ee942e16fdef6c
Sep 16 18:51:58.895220 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.895195    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-7jnv6"]
Sep 16 18:51:58.977767 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:58.977735    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:59.012196 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.012168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c72322-36a0-4841-aeb4-e6156d441e1f-cert\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:59.014520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.014500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a3c72322-36a0-4841-aeb4-e6156d441e1f-cert\") pod \"llmisvc-controller-manager-c98cf57d5-wmmtx\" (UID: \"a3c72322-36a0-4841-aeb4-e6156d441e1f\") " pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:59.065635 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.065602    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:51:59.234361 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.234280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:51:59.513694 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.513658    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"]
Sep 16 18:51:59.516026 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:51:59.515994    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda3c72322_36a0_4841_aeb4_e6156d441e1f.slice/crio-36c28c86cfa69ec2ae7bb9d7a1197c14f227eea12c1678776cc7be111f2e83f0 WatchSource:0}: Error finding container 36c28c86cfa69ec2ae7bb9d7a1197c14f227eea12c1678776cc7be111f2e83f0: Status 404 returned error can't find the container with id 36c28c86cfa69ec2ae7bb9d7a1197c14f227eea12c1678776cc7be111f2e83f0
Sep 16 18:51:59.768266 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.768154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6" event={"ID":"49606616-60a9-4e85-9e9c-5e2a2b13976d","Type":"ContainerStarted","Data":"b1f7e2e8c176060026f88103913afee46d9db6514188aac2f0ee942e16fdef6c"}
Sep 16 18:51:59.769799 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:51:59.769764    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx" event={"ID":"a3c72322-36a0-4841-aeb4-e6156d441e1f","Type":"ContainerStarted","Data":"36c28c86cfa69ec2ae7bb9d7a1197c14f227eea12c1678776cc7be111f2e83f0"}
Sep 16 18:52:01.783073 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:01.783034    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6" event={"ID":"49606616-60a9-4e85-9e9c-5e2a2b13976d","Type":"ContainerStarted","Data":"0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1"}
Sep 16 18:52:01.783513 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:01.783182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:52:01.805681 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:01.805622    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6" podStartSLOduration=1.473656484 podStartE2EDuration="3.805603147s" totalImagesPullingTime="2.331946663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:01.802607753 +0000 UTC m=+505.914404394" watchObservedRunningTime="2026-09-16 18:52:01.805603147 +0000 UTC m=+505.917399790"
Sep 16 18:52:03.800587 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:03.800547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx" event={"ID":"a3c72322-36a0-4841-aeb4-e6156d441e1f","Type":"ContainerStarted","Data":"5b0920c2cc77f79916c0a6b00353a3953e7610188ea143d858b9e3c34d875c6d"}
Sep 16 18:52:03.801012 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:03.800614    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:52:03.836258 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:03.836208    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx" podStartSLOduration=2.29520434 podStartE2EDuration="5.836193929s" totalImagesPullingTime="3.540989589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:03.834540516 +0000 UTC m=+507.946337157" watchObservedRunningTime="2026-09-16 18:52:03.836193929 +0000 UTC m=+507.947990570"
Sep 16 18:52:32.799872 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:32.799797    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:52:33.120852 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:33.120760    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.248129 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:34.248095    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:52:34.811451 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:34.811418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-c98cf57d5-wmmtx"
Sep 16 18:52:42.986614 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:42.986575    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-7jnv6"]
Sep 16 18:52:42.987175 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:42.986793    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6" podUID="49606616-60a9-4e85-9e9c-5e2a2b13976d" containerName="manager" containerID="cri-o://0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1" gracePeriod=10
Sep 16 18:52:43.009655 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.009631    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-cjr45"]
Sep 16 18:52:43.012074 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.012059    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.022403 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.022382    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-cjr45"]
Sep 16 18:52:43.056719 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.056687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78-cert\") pod \"kserve-controller-manager-f8c46b677-cjr45\" (UID: \"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78\") " pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.056893 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.056730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q2gb\" (UniqueName: \"kubernetes.io/projected/2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78-kube-api-access-9q2gb\") pod \"kserve-controller-manager-f8c46b677-cjr45\" (UID: \"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78\") " pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.157486 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.157450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78-cert\") pod \"kserve-controller-manager-f8c46b677-cjr45\" (UID: \"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78\") " pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.157619 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.157498    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q2gb\" (UniqueName: \"kubernetes.io/projected/2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78-kube-api-access-9q2gb\") pod \"kserve-controller-manager-f8c46b677-cjr45\" (UID: \"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78\") " pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.159899 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.159845    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78-cert\") pod \"kserve-controller-manager-f8c46b677-cjr45\" (UID: \"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78\") " pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.166931 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.166906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q2gb\" (UniqueName: \"kubernetes.io/projected/2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78-kube-api-access-9q2gb\") pod \"kserve-controller-manager-f8c46b677-cjr45\" (UID: \"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78\") " pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.230072 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.230047    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:52:43.258945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.258849    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bc74\" (UniqueName: \"kubernetes.io/projected/49606616-60a9-4e85-9e9c-5e2a2b13976d-kube-api-access-7bc74\") pod \"49606616-60a9-4e85-9e9c-5e2a2b13976d\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") "
Sep 16 18:52:43.259102 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.258945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49606616-60a9-4e85-9e9c-5e2a2b13976d-cert\") pod \"49606616-60a9-4e85-9e9c-5e2a2b13976d\" (UID: \"49606616-60a9-4e85-9e9c-5e2a2b13976d\") "
Sep 16 18:52:43.261742 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.261709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49606616-60a9-4e85-9e9c-5e2a2b13976d-cert" (OuterVolumeSpecName: "cert") pod "49606616-60a9-4e85-9e9c-5e2a2b13976d" (UID: "49606616-60a9-4e85-9e9c-5e2a2b13976d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:52:43.261889 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.261745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49606616-60a9-4e85-9e9c-5e2a2b13976d-kube-api-access-7bc74" (OuterVolumeSpecName: "kube-api-access-7bc74") pod "49606616-60a9-4e85-9e9c-5e2a2b13976d" (UID: "49606616-60a9-4e85-9e9c-5e2a2b13976d"). InnerVolumeSpecName "kube-api-access-7bc74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:52:43.360118 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.360084    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49606616-60a9-4e85-9e9c-5e2a2b13976d-cert\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:52:43.360118 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.360115    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bc74\" (UniqueName: \"kubernetes.io/projected/49606616-60a9-4e85-9e9c-5e2a2b13976d-kube-api-access-7bc74\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:52:43.371086 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.371051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:43.654507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:43.654473    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-cjr45"]
Sep 16 18:52:43.657055 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:52:43.657026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2409e2cf_a6d0_43e2_b63f_5fe61d6f6e78.slice/crio-6d9a3fa6980d14b3bf2e62fb961cfc66a5ca3488bb76531e46b6bfe40c3f1b3d WatchSource:0}: Error finding container 6d9a3fa6980d14b3bf2e62fb961cfc66a5ca3488bb76531e46b6bfe40c3f1b3d: Status 404 returned error can't find the container with id 6d9a3fa6980d14b3bf2e62fb961cfc66a5ca3488bb76531e46b6bfe40c3f1b3d
Sep 16 18:52:44.009165 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.009136    2566 generic.go:356] "Generic (PLEG): container finished" podID="49606616-60a9-4e85-9e9c-5e2a2b13976d" containerID="0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1" exitCode=0
Sep 16 18:52:44.009601 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.009181    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6"
Sep 16 18:52:44.009601 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.009224    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6" event={"ID":"49606616-60a9-4e85-9e9c-5e2a2b13976d","Type":"ContainerDied","Data":"0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1"}
Sep 16 18:52:44.009601 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.009284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-7jnv6" event={"ID":"49606616-60a9-4e85-9e9c-5e2a2b13976d","Type":"ContainerDied","Data":"b1f7e2e8c176060026f88103913afee46d9db6514188aac2f0ee942e16fdef6c"}
Sep 16 18:52:44.009601 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.009318    2566 scope.go:117] "RemoveContainer" containerID="0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1"
Sep 16 18:52:44.010632 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.010553    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-cjr45" event={"ID":"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78","Type":"ContainerStarted","Data":"6d9a3fa6980d14b3bf2e62fb961cfc66a5ca3488bb76531e46b6bfe40c3f1b3d"}
Sep 16 18:52:44.022100 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.022084    2566 scope.go:117] "RemoveContainer" containerID="0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1"
Sep 16 18:52:44.022361 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:52:44.022344    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1\": container with ID starting with 0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1 not found: ID does not exist" containerID="0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1"
Sep 16 18:52:44.022438 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.022367    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1"} err="failed to get container status \"0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1\": rpc error: code = NotFound desc = could not find container \"0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1\": container with ID starting with 0a198f1cb05c64e05e3c0910da4abebb0a031c42a8b2b25408acc62a4b7e31a1 not found: ID does not exist"
Sep 16 18:52:44.033749 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.033728    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-7jnv6"]
Sep 16 18:52:44.045992 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.045971    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-f8c46b677-7jnv6"]
Sep 16 18:52:44.402613 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:44.402580    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49606616-60a9-4e85-9e9c-5e2a2b13976d" path="/var/lib/kubelet/pods/49606616-60a9-4e85-9e9c-5e2a2b13976d/volumes"
Sep 16 18:52:45.018232 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:45.018200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-f8c46b677-cjr45" event={"ID":"2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78","Type":"ContainerStarted","Data":"209ae00eff55cc24668fc4d12098a599df6af77a010e0a428e815c3b8f2cf349"}
Sep 16 18:52:45.018641 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:45.018302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:52:45.037036 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:45.036991    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-f8c46b677-cjr45" podStartSLOduration=2.668067877 podStartE2EDuration="3.036977034s" totalImagesPullingTime="368.909157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:45.035032454 +0000 UTC m=+549.146829095" watchObservedRunningTime="2026-09-16 18:52:45.036977034 +0000 UTC m=+549.148773675"
Sep 16 18:52:52.474633 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:52:52.474596    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:16.029380 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:16.029344    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-f8c46b677-cjr45"
Sep 16 18:53:17.302842 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.302797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.351422 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.351380    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-rr9qw"]
Sep 16 18:53:17.351917 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.351901    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49606616-60a9-4e85-9e9c-5e2a2b13976d" containerName="manager"
Sep 16 18:53:17.351970 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.351920    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="49606616-60a9-4e85-9e9c-5e2a2b13976d" containerName="manager"
Sep 16 18:53:17.352023 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.352012    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="49606616-60a9-4e85-9e9c-5e2a2b13976d" containerName="manager"
Sep 16 18:53:17.355520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.355497    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.358614 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.358589    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 16 18:53:17.358737 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.358594    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-zgqpf\""
Sep 16 18:53:17.358974 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.358953    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 16 18:53:17.363987 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.363965    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-rr9qw"]
Sep 16 18:53:17.422177 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.422142    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.439189 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.439149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-cert\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.439386 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.439302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-metrics-certs\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.439386 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.439350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v72f\" (UniqueName: \"kubernetes.io/projected/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-kube-api-access-6v72f\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.540391 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.540349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-cert\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.540604 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.540426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-metrics-certs\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.540604 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.540461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v72f\" (UniqueName: \"kubernetes.io/projected/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-kube-api-access-6v72f\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.540604 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:53:17.540578    2566 secret.go:189] Couldn't get secret kserve/odh-model-controller-metrics-tls: secret "odh-model-controller-metrics-tls" not found
Sep 16 18:53:17.540783 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:53:17.540688    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-metrics-certs podName:f2d8b15d-8a41-4b4d-ac24-36c411cc6a60 nodeName:}" failed. No retries permitted until 2026-09-16 18:53:18.040662987 +0000 UTC m=+582.152459609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-metrics-certs") pod "odh-model-controller-7b966d7984-rr9qw" (UID: "f2d8b15d-8a41-4b4d-ac24-36c411cc6a60") : secret "odh-model-controller-metrics-tls" not found
Sep 16 18:53:17.543072 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.543051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-cert\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.552188 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.552160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v72f\" (UniqueName: \"kubernetes.io/projected/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-kube-api-access-6v72f\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:17.567339 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.567260    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.675607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.675566    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:17.853731 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:17.853629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:18.045453 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:18.045414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-metrics-certs\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:18.047783 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:18.047764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d8b15d-8a41-4b4d-ac24-36c411cc6a60-metrics-certs\") pod \"odh-model-controller-7b966d7984-rr9qw\" (UID: \"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60\") " pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:18.268764 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:18.268722    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:18.426084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:18.426047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:18.484280 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:18.484241    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:53:18.540979 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:18.540953    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-rr9qw"]
Sep 16 18:53:18.542555 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:53:18.542530    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d8b15d_8a41_4b4d_ac24_36c411cc6a60.slice/crio-ed4a46d61b3a87258bd3746a9006947d09bf04135e07751672db8da146141b4a WatchSource:0}: Error finding container ed4a46d61b3a87258bd3746a9006947d09bf04135e07751672db8da146141b4a: Status 404 returned error can't find the container with id ed4a46d61b3a87258bd3746a9006947d09bf04135e07751672db8da146141b4a
Sep 16 18:53:19.194204 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:19.194157    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-rr9qw" event={"ID":"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60","Type":"ContainerStarted","Data":"ed4a46d61b3a87258bd3746a9006947d09bf04135e07751672db8da146141b4a"}
Sep 16 18:53:21.208550 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:21.208505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-rr9qw" event={"ID":"f2d8b15d-8a41-4b4d-ac24-36c411cc6a60","Type":"ContainerStarted","Data":"1f333f4a26f44640ff7a22c5bd38126d39654a71ff1dfd3a16f8a1251977afe8"}
Sep 16 18:53:21.209090 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:21.208599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:53:21.227075 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:21.227022    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-rr9qw" podStartSLOduration=1.68607237 podStartE2EDuration="4.227005897s" totalImagesPullingTime="2.540933527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:21.226085387 +0000 UTC m=+585.337882027" watchObservedRunningTime="2026-09-16 18:53:21.227005897 +0000 UTC m=+585.338802517"
Sep 16 18:53:32.216662 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:53:32.216630    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-rr9qw"
Sep 16 18:54:00.405896 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:54:00.405781    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:52.123233 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:54:52.123193    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:54:52.229582 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:54:52.229545    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:04.168436 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:04.168401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:04.363223 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:04.363193    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:22.326406 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.326367    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:22.549285 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.549224    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:22.674711 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.674664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:22.820693 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.820659    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"]
Sep 16 18:55:22.824733 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.824711    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:22.827687 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.827661    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"openshift-service-ca.crt\""
Sep 16 18:55:22.827687 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.827685    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"kube-root-ca.crt\""
Sep 16 18:55:22.829045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.829025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"default-dockercfg-v62w5\""
Sep 16 18:55:22.829182 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.829163    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Sep 16 18:55:22.829313 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.829297    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-63a03ef4\"/\"llmisvc-model-fb-opt-125m-route-dc21cb14-epp-sa-dockercfg-nbtbk\""
Sep 16 18:55:22.835407 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.835374    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"]
Sep 16 18:55:22.854813 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.854782    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:22.898309 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.898276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f5840-cd91-4e65-a795-a23619354a24-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:22.898516 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.898327    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68xxc\" (UniqueName: \"kubernetes.io/projected/982f5840-cd91-4e65-a795-a23619354a24-kube-api-access-68xxc\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:22.999045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.998963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68xxc\" (UniqueName: \"kubernetes.io/projected/982f5840-cd91-4e65-a795-a23619354a24-kube-api-access-68xxc\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:22.999187 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:22.999093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f5840-cd91-4e65-a795-a23619354a24-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:23.001642 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:23.001621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f5840-cd91-4e65-a795-a23619354a24-tls-certs\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:23.007071 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:23.007011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68xxc\" (UniqueName: \"kubernetes.io/projected/982f5840-cd91-4e65-a795-a23619354a24-kube-api-access-68xxc\") pod \"llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") " pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:23.143767 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:23.143723    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:23.419006 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:23.418980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"]
Sep 16 18:55:23.420202 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:55:23.420167    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982f5840_cd91_4e65_a795_a23619354a24.slice/crio-c49ca77c0ec0c595c632c064c709b080a3138fb6d07ef5ea1bcd554fd01b9583 WatchSource:0}: Error finding container c49ca77c0ec0c595c632c064c709b080a3138fb6d07ef5ea1bcd554fd01b9583: Status 404 returned error can't find the container with id c49ca77c0ec0c595c632c064c709b080a3138fb6d07ef5ea1bcd554fd01b9583
Sep 16 18:55:23.422368 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:23.422348    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:55:23.828351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:23.828269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" event={"ID":"982f5840-cd91-4e65-a795-a23619354a24","Type":"ContainerStarted","Data":"c49ca77c0ec0c595c632c064c709b080a3138fb6d07ef5ea1bcd554fd01b9583"}
Sep 16 18:55:26.849396 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:26.849363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" event={"ID":"982f5840-cd91-4e65-a795-a23619354a24","Type":"ContainerStarted","Data":"d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573"}
Sep 16 18:55:26.849784 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:26.849550    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:55:26.882910 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:26.882846    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" podStartSLOduration=2.498077132 podStartE2EDuration="4.882830572s" totalImagesPullingTime="2.38475344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:26.880300191 +0000 UTC m=+710.992096843" watchObservedRunningTime="2026-09-16 18:55:26.882830572 +0000 UTC m=+710.994627213"
Sep 16 18:55:29.191712 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.191639    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"]
Sep 16 18:55:29.194933 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.194915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.197739 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.197711    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"kube-root-ca.crt\""
Sep 16 18:55:29.198749 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.198730    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"fc-smoke-test-kserve-self-signed-certs\""
Sep 16 18:55:29.198867 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.198775    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"openshift-service-ca.crt\""
Sep 16 18:55:29.198867 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.198777    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-4f8df83c\"/\"default-dockercfg-jpgd9\""
Sep 16 18:55:29.209174 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.209147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"]
Sep 16 18:55:29.231573 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.231536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:29.257402 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.257376    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-home\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.257520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.257413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-dshm\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.257520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.257433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.257520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.257458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6996\" (UniqueName: \"kubernetes.io/projected/61f78393-768a-43cd-962b-3e504fbcde38-kube-api-access-n6996\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.257520 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.257487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f78393-768a-43cd-962b-3e504fbcde38-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.257680 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.257525    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.358734 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.358702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-home\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.358870 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.358740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-dshm\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.358870 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.358760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.358995 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.358903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6996\" (UniqueName: \"kubernetes.io/projected/61f78393-768a-43cd-962b-3e504fbcde38-kube-api-access-n6996\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.358995 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.358947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f78393-768a-43cd-962b-3e504fbcde38-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.359079 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.359016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.359232 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.359214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-tmp-dir\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.359333 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.359313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-home\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.359396 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.359363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-model-cache\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.361076 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.361054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-dshm\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.361399 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.361379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f78393-768a-43cd-962b-3e504fbcde38-tls-certs\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.366559 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.366539    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6996\" (UniqueName: \"kubernetes.io/projected/61f78393-768a-43cd-962b-3e504fbcde38-kube-api-access-n6996\") pod \"fc-smoke-test-kserve-7f4687897-fw56r\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") " pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.505340 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.505252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:29.547092 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.547055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:29.780954 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.780927    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"]
Sep 16 18:55:29.782702 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:55:29.782671    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61f78393_768a_43cd_962b_3e504fbcde38.slice/crio-733f89231eacca3a7b9e9654eaab07b8e0da569d82f6a7f37832ff5c7fc587d0 WatchSource:0}: Error finding container 733f89231eacca3a7b9e9654eaab07b8e0da569d82f6a7f37832ff5c7fc587d0: Status 404 returned error can't find the container with id 733f89231eacca3a7b9e9654eaab07b8e0da569d82f6a7f37832ff5c7fc587d0
Sep 16 18:55:29.865618 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:29.865573    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" event={"ID":"61f78393-768a-43cd-962b-3e504fbcde38","Type":"ContainerStarted","Data":"733f89231eacca3a7b9e9654eaab07b8e0da569d82f6a7f37832ff5c7fc587d0"}
Sep 16 18:55:31.877090 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:31.877049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" event={"ID":"61f78393-768a-43cd-962b-3e504fbcde38","Type":"ContainerStarted","Data":"3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37"}
Sep 16 18:55:31.896113 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:31.896057    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" podStartSLOduration=1.438774848 podStartE2EDuration="2.896041206s" totalImagesPullingTime="1.457266358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:31.8944169 +0000 UTC m=+716.006213542" watchObservedRunningTime="2026-09-16 18:55:31.896041206 +0000 UTC m=+716.007837847"
Sep 16 18:55:32.505470 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:32.505433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:39.261122 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:39.261079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:39.505504 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:39.505472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:39.518375 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:39.518301    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:39.935236 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:39.935202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:55:42.540871 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:42.540832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:55:57.857010 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:55:57.856975    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:56:11.839637 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:11.839585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:11.846519 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:11.846489    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"]
Sep 16 18:56:11.855569 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:56:11.855529    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:56:11.855569 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:11.855569 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:11.855569 ip-10-0-143-190 kubenswrapper[2566]:  > containerID="3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37" cmd=["/bin/sleep","15"]
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:11.855597    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" message="" err=<
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]:  >
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:56:11.855614    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" podUID="61f78393-768a-43cd-962b-3e504fbcde38" containerName="main" containerID="cri-o://3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37"
Sep 16 18:56:11.855900 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:11.855669    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" podUID="61f78393-768a-43cd-962b-3e504fbcde38" containerName="main" containerID="cri-o://3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37" gracePeriod=60
Sep 16 18:56:11.903573 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:11.903544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:56:11.924807 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:11.924775    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" podUID="61f78393-768a-43cd-962b-3e504fbcde38" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Sep 16 18:56:12.087426 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.087396    2566 generic.go:356] "Generic (PLEG): container finished" podID="61f78393-768a-43cd-962b-3e504fbcde38" containerID="3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37" exitCode=0
Sep 16 18:56:12.087673 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.087479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" event={"ID":"61f78393-768a-43cd-962b-3e504fbcde38","Type":"ContainerDied","Data":"3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37"}
Sep 16 18:56:12.202629 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.202609    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:56:12.326363 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326334    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-model-cache\") pod \"61f78393-768a-43cd-962b-3e504fbcde38\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") "
Sep 16 18:56:12.326363 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326365    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f78393-768a-43cd-962b-3e504fbcde38-tls-certs\") pod \"61f78393-768a-43cd-962b-3e504fbcde38\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") "
Sep 16 18:56:12.326571 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-tmp-dir\") pod \"61f78393-768a-43cd-962b-3e504fbcde38\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") "
Sep 16 18:56:12.326571 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326412    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-home\") pod \"61f78393-768a-43cd-962b-3e504fbcde38\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") "
Sep 16 18:56:12.326571 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-dshm\") pod \"61f78393-768a-43cd-962b-3e504fbcde38\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") "
Sep 16 18:56:12.326571 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6996\" (UniqueName: \"kubernetes.io/projected/61f78393-768a-43cd-962b-3e504fbcde38-kube-api-access-n6996\") pod \"61f78393-768a-43cd-962b-3e504fbcde38\" (UID: \"61f78393-768a-43cd-962b-3e504fbcde38\") "
Sep 16 18:56:12.326791 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326648    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-model-cache" (OuterVolumeSpecName: "model-cache") pod "61f78393-768a-43cd-962b-3e504fbcde38" (UID: "61f78393-768a-43cd-962b-3e504fbcde38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:12.326791 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "61f78393-768a-43cd-962b-3e504fbcde38" (UID: "61f78393-768a-43cd-962b-3e504fbcde38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:12.326791 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-home" (OuterVolumeSpecName: "home") pod "61f78393-768a-43cd-962b-3e504fbcde38" (UID: "61f78393-768a-43cd-962b-3e504fbcde38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:12.326953 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326805    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:56:12.326953 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326851    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:56:12.326953 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.326867    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:56:12.328679 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.328655    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-dshm" (OuterVolumeSpecName: "dshm") pod "61f78393-768a-43cd-962b-3e504fbcde38" (UID: "61f78393-768a-43cd-962b-3e504fbcde38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:12.328679 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.328670    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61f78393-768a-43cd-962b-3e504fbcde38-kube-api-access-n6996" (OuterVolumeSpecName: "kube-api-access-n6996") pod "61f78393-768a-43cd-962b-3e504fbcde38" (UID: "61f78393-768a-43cd-962b-3e504fbcde38"). InnerVolumeSpecName "kube-api-access-n6996". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:12.328786 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.328737    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61f78393-768a-43cd-962b-3e504fbcde38-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "61f78393-768a-43cd-962b-3e504fbcde38" (UID: "61f78393-768a-43cd-962b-3e504fbcde38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:12.427631 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.427602    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61f78393-768a-43cd-962b-3e504fbcde38-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:56:12.427758 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.427634    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61f78393-768a-43cd-962b-3e504fbcde38-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:56:12.427758 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:12.427649    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6996\" (UniqueName: \"kubernetes.io/projected/61f78393-768a-43cd-962b-3e504fbcde38-kube-api-access-n6996\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:56:13.093943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:13.093910    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"
Sep 16 18:56:13.093943 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:13.093921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r" event={"ID":"61f78393-768a-43cd-962b-3e504fbcde38","Type":"ContainerDied","Data":"733f89231eacca3a7b9e9654eaab07b8e0da569d82f6a7f37832ff5c7fc587d0"}
Sep 16 18:56:13.094482 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:13.093980    2566 scope.go:117] "RemoveContainer" containerID="3180621740fe20d2b75c63d103f584ea7073f9bdad1e8c6ece7850e16d585a37"
Sep 16 18:56:13.111778 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:13.111752    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"]
Sep 16 18:56:13.113858 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:13.113837    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-4f8df83c/fc-smoke-test-kserve-7f4687897-fw56r"]
Sep 16 18:56:14.403334 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:14.403300    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61f78393-768a-43cd-962b-3e504fbcde38" path="/var/lib/kubelet/pods/61f78393-768a-43cd-962b-3e504fbcde38/volumes"
Sep 16 18:56:55.909453 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:56:55.909415    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:01.896961 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:01.896850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:02.013141 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:02.013110    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:05.943541 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.943506    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"]
Sep 16 18:57:05.944020 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.943846    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61f78393-768a-43cd-962b-3e504fbcde38" containerName="main"
Sep 16 18:57:05.944020 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.943857    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="61f78393-768a-43cd-962b-3e504fbcde38" containerName="main"
Sep 16 18:57:05.944020 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.943951    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="61f78393-768a-43cd-962b-3e504fbcde38" containerName="main"
Sep 16 18:57:05.946218 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.946202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:05.949868 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.949833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"kube-root-ca.crt\""
Sep 16 18:57:05.951009 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.950994    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"default-dockercfg-c5ftv\""
Sep 16 18:57:05.951072 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.951020    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"openshift-service-ca.crt\""
Sep 16 18:57:05.951133 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.951029    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-flow-control-smoke-1e98f46d\"/\"fc-concurrency-test-kserve-self-signed-certs\""
Sep 16 18:57:05.960243 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.960220    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"]
Sep 16 18:57:05.965499 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:05.965471    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:06.081678 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.081650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.081843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.081699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.081843 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.081731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.082006 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.081871    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-home\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.082006 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.081942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.082006 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.081979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdc66\" (UniqueName: \"kubernetes.io/projected/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-kube-api-access-cdc66\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.182431 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.182399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdc66\" (UniqueName: \"kubernetes.io/projected/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-kube-api-access-cdc66\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.182594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.182443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.182594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.182483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.182594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.182508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.182594 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.182585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-home\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.183248 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.182610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.183248 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.183099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tmp-dir\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.183248 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.183212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-model-cache\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.183441 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.183419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-home\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.185228 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.185199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-dshm\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.185558 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.185541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tls-certs\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.211282 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.211207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdc66\" (UniqueName: \"kubernetes.io/projected/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-kube-api-access-cdc66\") pod \"fc-concurrency-test-kserve-b5d44467c-pd9rg\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") " pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.259945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.259906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:06.358991 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.358955    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:06.546859 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:06.546833    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"]
Sep 16 18:57:06.547959 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:57:06.547933    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f1a8aa_750a_4e47_9aa4_7251f63de9a6.slice/crio-5c47b3322489b748e805efd85c050b1b475946ddb9aefd9925dd9553c07b49d6 WatchSource:0}: Error finding container 5c47b3322489b748e805efd85c050b1b475946ddb9aefd9925dd9553c07b49d6: Status 404 returned error can't find the container with id 5c47b3322489b748e805efd85c050b1b475946ddb9aefd9925dd9553c07b49d6
Sep 16 18:57:07.369156 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:07.369123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" event={"ID":"88f1a8aa-750a-4e47-9aa4-7251f63de9a6","Type":"ContainerStarted","Data":"b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8"}
Sep 16 18:57:07.369619 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:07.369176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" event={"ID":"88f1a8aa-750a-4e47-9aa4-7251f63de9a6","Type":"ContainerStarted","Data":"5c47b3322489b748e805efd85c050b1b475946ddb9aefd9925dd9553c07b49d6"}
Sep 16 18:57:07.399649 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:07.399605    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" podStartSLOduration=2.399589837 podStartE2EDuration="2.399589837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:07.396802279 +0000 UTC m=+811.508598921" watchObservedRunningTime="2026-09-16 18:57:07.399589837 +0000 UTC m=+811.511386479"
Sep 16 18:57:08.260393 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:08.260351    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:09.533945 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:09.533913    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:12.258553 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:12.258514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:15.920344 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:15.920284    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:16.020893 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:16.020846    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:16.260329 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:16.260247    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:16.273045 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:16.273014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:16.428266 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:16.428231    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:16.772741 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:16.772703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:24.392869 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:24.392828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:39.845624 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:39.845583    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:39.864502 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:39.864470    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"]
Sep 16 18:57:39.921287 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:39.921251    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:45.054803 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.054768    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"]
Sep 16 18:57:45.063017 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.062978    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:45.065036 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:57:45.064999    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:57:45.065036 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:57:45.065036 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:57:45.065036 ip-10-0-143-190 kubenswrapper[2566]:  > containerID="b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8" cmd=["/bin/sleep","15"]
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.065049    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" message="" err=<
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]:  >
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:57:45.065061    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]:  > pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerName="main" containerID="cri-o://b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8"
Sep 16 18:57:45.065579 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.065106    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerName="main" containerID="cri-o://b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8" gracePeriod=60
Sep 16 18:57:45.154081 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.154043    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:57:45.418295 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.418251    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": dial tcp 10.133.0.47:8000: connect: connection refused"
Sep 16 18:57:45.563690 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.563654    2566 generic.go:356] "Generic (PLEG): container finished" podID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerID="b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8" exitCode=0
Sep 16 18:57:45.563859 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.563718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" event={"ID":"88f1a8aa-750a-4e47-9aa4-7251f63de9a6","Type":"ContainerDied","Data":"b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8"}
Sep 16 18:57:45.811278 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.811248    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:45.921629 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921600    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdc66\" (UniqueName: \"kubernetes.io/projected/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-kube-api-access-cdc66\") pod \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") "
Sep 16 18:57:45.921765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921666    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-home\") pod \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") "
Sep 16 18:57:45.921765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-model-cache\") pod \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") "
Sep 16 18:57:45.921765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921719    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tls-certs\") pod \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") "
Sep 16 18:57:45.921765 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921754    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tmp-dir\") pod \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") "
Sep 16 18:57:45.922138 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921807    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-dshm\") pod \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\" (UID: \"88f1a8aa-750a-4e47-9aa4-7251f63de9a6\") "
Sep 16 18:57:45.922138 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-home" (OuterVolumeSpecName: "home") pod "88f1a8aa-750a-4e47-9aa4-7251f63de9a6" (UID: "88f1a8aa-750a-4e47-9aa4-7251f63de9a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:45.922138 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.921988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-model-cache" (OuterVolumeSpecName: "model-cache") pod "88f1a8aa-750a-4e47-9aa4-7251f63de9a6" (UID: "88f1a8aa-750a-4e47-9aa4-7251f63de9a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:45.922284 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.922137    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88f1a8aa-750a-4e47-9aa4-7251f63de9a6" (UID: "88f1a8aa-750a-4e47-9aa4-7251f63de9a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:45.924059 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.924019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88f1a8aa-750a-4e47-9aa4-7251f63de9a6" (UID: "88f1a8aa-750a-4e47-9aa4-7251f63de9a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:57:45.924189 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.924107    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-dshm" (OuterVolumeSpecName: "dshm") pod "88f1a8aa-750a-4e47-9aa4-7251f63de9a6" (UID: "88f1a8aa-750a-4e47-9aa4-7251f63de9a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:57:45.924189 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:45.924122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-kube-api-access-cdc66" (OuterVolumeSpecName: "kube-api-access-cdc66") pod "88f1a8aa-750a-4e47-9aa4-7251f63de9a6" (UID: "88f1a8aa-750a-4e47-9aa4-7251f63de9a6"). InnerVolumeSpecName "kube-api-access-cdc66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:57:46.022667 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.022636    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.022667 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.022663    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdc66\" (UniqueName: \"kubernetes.io/projected/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-kube-api-access-cdc66\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.022667 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.022673    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.022872 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.022683    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.022872 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.022693    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.022872 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.022702    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88f1a8aa-750a-4e47-9aa4-7251f63de9a6-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:57:46.569926 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.569897    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"
Sep 16 18:57:46.570300 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.569871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg" event={"ID":"88f1a8aa-750a-4e47-9aa4-7251f63de9a6","Type":"ContainerDied","Data":"5c47b3322489b748e805efd85c050b1b475946ddb9aefd9925dd9553c07b49d6"}
Sep 16 18:57:46.570300 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.570032    2566 scope.go:117] "RemoveContainer" containerID="b077e2dbb26f11c1b1abcdc76a891d552a907ff4346fbccbd4873288d93d42b8"
Sep 16 18:57:46.587477 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.587455    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"]
Sep 16 18:57:46.594066 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:46.594044    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-flow-control-smoke-1e98f46d/fc-concurrency-test-kserve-b5d44467c-pd9rg"]
Sep 16 18:57:47.857112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:47.857072    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:57:48.402060 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:48.402031    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" path="/var/lib/kubelet/pods/88f1a8aa-750a-4e47-9aa4-7251f63de9a6/volumes"
Sep 16 18:57:54.885910 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:54.885858    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main" containerID="cri-o://d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573" gracePeriod=45
Sep 16 18:57:57.856699 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:57:57.856661    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:07.856381 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:07.856346    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:07.856750 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:07.856483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:58:17.856507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:17.856471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 18:58:25.129744 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.129719    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:58:25.223607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.223525    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68xxc\" (UniqueName: \"kubernetes.io/projected/982f5840-cd91-4e65-a795-a23619354a24-kube-api-access-68xxc\") pod \"982f5840-cd91-4e65-a795-a23619354a24\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") "
Sep 16 18:58:25.223607 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.223589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f5840-cd91-4e65-a795-a23619354a24-tls-certs\") pod \"982f5840-cd91-4e65-a795-a23619354a24\" (UID: \"982f5840-cd91-4e65-a795-a23619354a24\") "
Sep 16 18:58:25.225696 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.225668    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/982f5840-cd91-4e65-a795-a23619354a24-kube-api-access-68xxc" (OuterVolumeSpecName: "kube-api-access-68xxc") pod "982f5840-cd91-4e65-a795-a23619354a24" (UID: "982f5840-cd91-4e65-a795-a23619354a24"). InnerVolumeSpecName "kube-api-access-68xxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:25.225696 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.225692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/982f5840-cd91-4e65-a795-a23619354a24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "982f5840-cd91-4e65-a795-a23619354a24" (UID: "982f5840-cd91-4e65-a795-a23619354a24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:25.324385 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.324360    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982f5840-cd91-4e65-a795-a23619354a24-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:25.324385 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.324382    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68xxc\" (UniqueName: \"kubernetes.io/projected/982f5840-cd91-4e65-a795-a23619354a24-kube-api-access-68xxc\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:58:25.765012 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.764979    2566 generic.go:356] "Generic (PLEG): container finished" podID="982f5840-cd91-4e65-a795-a23619354a24" containerID="d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573" exitCode=0
Sep 16 18:58:25.765215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.765024    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"
Sep 16 18:58:25.765215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.765074    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" event={"ID":"982f5840-cd91-4e65-a795-a23619354a24","Type":"ContainerDied","Data":"d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573"}
Sep 16 18:58:25.765215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.765137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2" event={"ID":"982f5840-cd91-4e65-a795-a23619354a24","Type":"ContainerDied","Data":"c49ca77c0ec0c595c632c064c709b080a3138fb6d07ef5ea1bcd554fd01b9583"}
Sep 16 18:58:25.765215 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.765165    2566 scope.go:117] "RemoveContainer" containerID="d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573"
Sep 16 18:58:25.774799 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.774783    2566 scope.go:117] "RemoveContainer" containerID="d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573"
Sep 16 18:58:25.775108 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:58:25.775090    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573\": container with ID starting with d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573 not found: ID does not exist" containerID="d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573"
Sep 16 18:58:25.775180 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.775116    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573"} err="failed to get container status \"d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573\": rpc error: code = NotFound desc = could not find container \"d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573\": container with ID starting with d322555848b07a66427644360285593a5bdaa40c00bf6a3cfc2fb82ff632d573 not found: ID does not exist"
Sep 16 18:58:25.787304 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.787282    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"]
Sep 16 18:58:25.791201 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:25.791182    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-63a03ef4/llmisvc32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-router-scheqdrr2"]
Sep 16 18:58:26.402391 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:26.402357    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="982f5840-cd91-4e65-a795-a23619354a24" path="/var/lib/kubelet/pods/982f5840-cd91-4e65-a795-a23619354a24/volumes"
Sep 16 18:58:29.048351 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:29.048263    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:29.147465 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:29.147432    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:34.998628 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.998597    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"]
Sep 16 18:58:34.998994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.998972    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main"
Sep 16 18:58:34.998994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.998986    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main"
Sep 16 18:58:34.998994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.998994    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerName="main"
Sep 16 18:58:34.999100 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.998999    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerName="main"
Sep 16 18:58:34.999100 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.999062    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="982f5840-cd91-4e65-a795-a23619354a24" containerName="main"
Sep 16 18:58:34.999100 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:34.999072    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f1a8aa-750a-4e47-9aa4-7251f63de9a6" containerName="main"
Sep 16 18:58:35.001938 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.001920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.004826 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.004804    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"scheduler-inline-config-test-kserve-self-signed-certs\""
Sep 16 18:58:35.005994 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.005976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"kube-root-ca.crt\""
Sep 16 18:58:35.006108 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.005976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"openshift-service-ca.crt\""
Sep 16 18:58:35.006108 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.006039    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-47655ff5\"/\"default-dockercfg-l9hdm\""
Sep 16 18:58:35.018683 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.018659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"]
Sep 16 18:58:35.035980 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.035954    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.108585 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.108555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.108720 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.108595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtd98\" (UniqueName: \"kubernetes.io/projected/e6cedeae-cee4-44da-b1b2-1512f906c318-kube-api-access-dtd98\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.108720 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.108625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cedeae-cee4-44da-b1b2-1512f906c318-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.108854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.108765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.108854 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.108831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.108989 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.108860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.111596 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.111573    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.209451 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.209590 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtd98\" (UniqueName: \"kubernetes.io/projected/e6cedeae-cee4-44da-b1b2-1512f906c318-kube-api-access-dtd98\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.209590 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cedeae-cee4-44da-b1b2-1512f906c318-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.209713 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.209770 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209756    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.209829 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209789    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.209912 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.209829    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-model-cache\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.210093 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.210073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-home\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.210153 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.210135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-tmp-dir\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.211974 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.211944    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.212267 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.212241    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-dshm\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.212426 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.212404    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cedeae-cee4-44da-b1b2-1512f906c318-tls-certs\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.223064 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.223040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtd98\" (UniqueName: \"kubernetes.io/projected/e6cedeae-cee4-44da-b1b2-1512f906c318-kube-api-access-dtd98\") pod \"scheduler-inline-config-test-kserve-8655bc757-8pxvx\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") " pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.317313 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.317263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:35.352276 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.352193    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.452951 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.452919    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.594337 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.594312    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"]
Sep 16 18:58:35.595605 ip-10-0-143-190 kubenswrapper[2566]: W0916 18:58:35.595581    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cedeae_cee4_44da_b1b2_1512f906c318.slice/crio-74f5b2804726b9095f313551562c7ace0ec48a59835507b10acf34155fdd46d9 WatchSource:0}: Error finding container 74f5b2804726b9095f313551562c7ace0ec48a59835507b10acf34155fdd46d9: Status 404 returned error can't find the container with id 74f5b2804726b9095f313551562c7ace0ec48a59835507b10acf34155fdd46d9
Sep 16 18:58:35.695763 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.695730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:35.819275 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.819234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" event={"ID":"e6cedeae-cee4-44da-b1b2-1512f906c318","Type":"ContainerStarted","Data":"7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8"}
Sep 16 18:58:35.819452 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.819286    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" event={"ID":"e6cedeae-cee4-44da-b1b2-1512f906c318","Type":"ContainerStarted","Data":"74f5b2804726b9095f313551562c7ace0ec48a59835507b10acf34155fdd46d9"}
Sep 16 18:58:35.839657 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:35.839604    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" podStartSLOduration=1.839588539 podStartE2EDuration="1.839588539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:35.83803026 +0000 UTC m=+899.949826902" watchObservedRunningTime="2026-09-16 18:58:35.839588539 +0000 UTC m=+899.951385179"
Sep 16 18:58:36.318041 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:36.318013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:37.344927 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:37.344875    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:39.018019 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:39.017972    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:41.762307 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:41.762267    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:43.465145 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:43.465104    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:43.566605 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:43.566576    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:45.317507 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:45.317473    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:45.329694 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:45.329670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:45.882450 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:45.882417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:58:47.877363 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:47.877325    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:47.978161 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:47.978121    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:58:55.944571 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:58:55.944531    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:05.747117 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:05.747065    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:21.657983 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:21.657937    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:23.426549 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:23.426513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:23.528752 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:23.528713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:23.634659 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:23.634616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:23.791369 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:23.791332    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:23.892112 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:23.892070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:26.732925 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:26.732894    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:28.112899 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:28.112845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:28.512821 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:28.512785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:33.608846 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:33.608805    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:35.155165 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:35.155129    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:40.095699 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:40.095663    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.767078 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:41.767038    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:41.886363 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:41.886327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:42.583438 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:42.583406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:43.848388 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:43.848348    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:45.817848 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:45.817814    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:45.837756 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:45.837711    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"]
Sep 16 18:59:45.849823 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:59:45.849790    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 18:59:45.849823 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:45.849823 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:45.849823 ip-10-0-143-190 kubenswrapper[2566]:  > containerID="7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8" cmd=["/bin/sleep","15"]
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:45.849838    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" message="" err=<
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]:  >
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]: E0916 18:59:45.849850    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerName="main" containerID="cri-o://7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8"
Sep 16 18:59:45.850108 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:45.849921    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerName="main" containerID="cri-o://7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8" gracePeriod=60
Sep 16 18:59:45.872691 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:45.872661    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Sep 16 18:59:45.918310 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:45.918278    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:46.017985 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.017951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:46.182926 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.182871    2566 generic.go:356] "Generic (PLEG): container finished" podID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerID="7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8" exitCode=0
Sep 16 18:59:46.183048 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.182950    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" event={"ID":"e6cedeae-cee4-44da-b1b2-1512f906c318","Type":"ContainerDied","Data":"7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8"}
Sep 16 18:59:46.200330 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.200309    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:59:46.278043 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-home\") pod \"e6cedeae-cee4-44da-b1b2-1512f906c318\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") "
Sep 16 18:59:46.278206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-tmp-dir\") pod \"e6cedeae-cee4-44da-b1b2-1512f906c318\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") "
Sep 16 18:59:46.278206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278089    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-model-cache\") pod \"e6cedeae-cee4-44da-b1b2-1512f906c318\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") "
Sep 16 18:59:46.278206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278107    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-dshm\") pod \"e6cedeae-cee4-44da-b1b2-1512f906c318\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") "
Sep 16 18:59:46.278206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cedeae-cee4-44da-b1b2-1512f906c318-tls-certs\") pod \"e6cedeae-cee4-44da-b1b2-1512f906c318\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") "
Sep 16 18:59:46.278206 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278197    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtd98\" (UniqueName: \"kubernetes.io/projected/e6cedeae-cee4-44da-b1b2-1512f906c318-kube-api-access-dtd98\") pod \"e6cedeae-cee4-44da-b1b2-1512f906c318\" (UID: \"e6cedeae-cee4-44da-b1b2-1512f906c318\") "
Sep 16 18:59:46.278456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278329    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e6cedeae-cee4-44da-b1b2-1512f906c318" (UID: "e6cedeae-cee4-44da-b1b2-1512f906c318"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:46.278456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-home" (OuterVolumeSpecName: "home") pod "e6cedeae-cee4-44da-b1b2-1512f906c318" (UID: "e6cedeae-cee4-44da-b1b2-1512f906c318"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:46.278456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278354    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-model-cache" (OuterVolumeSpecName: "model-cache") pod "e6cedeae-cee4-44da-b1b2-1512f906c318" (UID: "e6cedeae-cee4-44da-b1b2-1512f906c318"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:46.278456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278420    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:59:46.278456 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278451    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:59:46.278633 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.278462    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:59:46.280376 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.280352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-dshm" (OuterVolumeSpecName: "dshm") pod "e6cedeae-cee4-44da-b1b2-1512f906c318" (UID: "e6cedeae-cee4-44da-b1b2-1512f906c318"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:46.280479 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.280382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6cedeae-cee4-44da-b1b2-1512f906c318-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e6cedeae-cee4-44da-b1b2-1512f906c318" (UID: "e6cedeae-cee4-44da-b1b2-1512f906c318"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:46.280479 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.280431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cedeae-cee4-44da-b1b2-1512f906c318-kube-api-access-dtd98" (OuterVolumeSpecName: "kube-api-access-dtd98") pod "e6cedeae-cee4-44da-b1b2-1512f906c318" (UID: "e6cedeae-cee4-44da-b1b2-1512f906c318"). InnerVolumeSpecName "kube-api-access-dtd98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:46.379084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.379009    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6cedeae-cee4-44da-b1b2-1512f906c318-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:59:46.379084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.379033    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6cedeae-cee4-44da-b1b2-1512f906c318-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:59:46.379084 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.379045    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtd98\" (UniqueName: \"kubernetes.io/projected/e6cedeae-cee4-44da-b1b2-1512f906c318-kube-api-access-dtd98\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 18:59:46.836701 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:46.836668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:47.189818 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:47.189786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"
Sep 16 18:59:47.190018 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:47.189825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx" event={"ID":"e6cedeae-cee4-44da-b1b2-1512f906c318","Type":"ContainerDied","Data":"74f5b2804726b9095f313551562c7ace0ec48a59835507b10acf34155fdd46d9"}
Sep 16 18:59:47.190018 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:47.189903    2566 scope.go:117] "RemoveContainer" containerID="7616003a3e59a9e28c422b359f0f4beeb9c4ca3a861005cd188b3b973ff8e5f8"
Sep 16 18:59:47.230466 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:47.230439    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"]
Sep 16 18:59:47.236819 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:47.236795    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-47655ff5/scheduler-inline-config-test-kserve-8655bc757-8pxvx"]
Sep 16 18:59:48.402458 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:48.402413    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" path="/var/lib/kubelet/pods/e6cedeae-cee4-44da-b1b2-1512f906c318/volumes"
Sep 16 18:59:50.911472 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:50.911420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:56.136266 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:56.136237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 18:59:57.004210 ip-10-0-143-190 kubenswrapper[2566]: I0916 18:59:57.004173    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:02.822306 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:02.822225    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:05.418607 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.418565    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:05.651097 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.651067    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"]
Sep 16 19:00:05.651441 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.651422    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerName="main"
Sep 16 19:00:05.651441 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.651440    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerName="main"
Sep 16 19:00:05.651613 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.651499    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6cedeae-cee4-44da-b1b2-1512f906c318" containerName="main"
Sep 16 19:00:05.654622 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.654600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.658701 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.658680    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"kube-root-ca.crt\""
Sep 16 19:00:05.658701 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.658703    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-epp-sa-dockercfg-jm5hq\""
Sep 16 19:00:05.658910 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.658732    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"default-dockercfg-hn45b\""
Sep 16 19:00:05.658910 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.658712    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"auth-enabled-test-kserve-self-signed-certs\""
Sep 16 19:00:05.658910 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.658798    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-enabled-requires-token-d4ed2c08\"/\"openshift-service-ca.crt\""
Sep 16 19:00:05.664279 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.664256    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"]
Sep 16 19:00:05.728413 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.728345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934c1f55-ba21-440d-a986-3300e64fff6d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.728526 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.728405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbkvq\" (UniqueName: \"kubernetes.io/projected/934c1f55-ba21-440d-a986-3300e64fff6d-kube-api-access-rbkvq\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.829463 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.829431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934c1f55-ba21-440d-a986-3300e64fff6d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.829622 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.829473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbkvq\" (UniqueName: \"kubernetes.io/projected/934c1f55-ba21-440d-a986-3300e64fff6d-kube-api-access-rbkvq\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.831915 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.831875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934c1f55-ba21-440d-a986-3300e64fff6d-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.837674 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.837645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbkvq\" (UniqueName: \"kubernetes.io/projected/934c1f55-ba21-440d-a986-3300e64fff6d-kube-api-access-rbkvq\") pod \"auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") " pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:05.964826 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:05.964788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:06.010079 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:06.009924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:06.235123 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:06.235095    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"]
Sep 16 19:00:06.236628 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:00:06.236601    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod934c1f55_ba21_440d_a986_3300e64fff6d.slice/crio-d5f697857d73be6e2f9021790d371e0536c8f24ed67d049adc568db6c5a2aad9 WatchSource:0}: Error finding container d5f697857d73be6e2f9021790d371e0536c8f24ed67d049adc568db6c5a2aad9: Status 404 returned error can't find the container with id d5f697857d73be6e2f9021790d371e0536c8f24ed67d049adc568db6c5a2aad9
Sep 16 19:00:06.293953 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:06.293924    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" event={"ID":"934c1f55-ba21-440d-a986-3300e64fff6d","Type":"ContainerStarted","Data":"d5f697857d73be6e2f9021790d371e0536c8f24ed67d049adc568db6c5a2aad9"}
Sep 16 19:00:07.299669 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:07.299631    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" event={"ID":"934c1f55-ba21-440d-a986-3300e64fff6d","Type":"ContainerStarted","Data":"f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a"}
Sep 16 19:00:07.300167 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:07.299748    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:07.319662 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:07.319595    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" podStartSLOduration=2.319582228 podStartE2EDuration="2.319582228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:07.31793203 +0000 UTC m=+991.429728671" watchObservedRunningTime="2026-09-16 19:00:07.319582228 +0000 UTC m=+991.431378867"
Sep 16 19:00:11.970343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:11.970301    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:12.384786 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:12.384688    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:18.632347 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:18.632306    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:33.325549 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:33.325506    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:33.422488 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:33.422448    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:38.306613 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:38.306587    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:00:40.594847 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:40.594809    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:47.499258 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:47.499197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:47.611312 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:47.611272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:48.369659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.369627    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:48.583116 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.583084    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"]
Sep 16 19:00:48.587617 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.587597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.591134 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.591110    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"openshift-service-ca.crt\""
Sep 16 19:00:48.592305 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.592286    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"kube-root-ca.crt\""
Sep 16 19:00:48.592425 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.592306    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Sep 16 19:00:48.592536 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.592520    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"default-dockercfg-hfdsq\""
Sep 16 19:00:48.592655 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.592634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-aae51b37\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-4fzr6\""
Sep 16 19:00:48.604361 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.604340    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"]
Sep 16 19:00:48.674775 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.674743    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8g9\" (UniqueName: \"kubernetes.io/projected/37230471-1ae4-4696-b8c8-8c3c165dae31-kube-api-access-zn8g9\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.674968 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.674786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37230471-1ae4-4696-b8c8-8c3c165dae31-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.776016 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.775987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8g9\" (UniqueName: \"kubernetes.io/projected/37230471-1ae4-4696-b8c8-8c3c165dae31-kube-api-access-zn8g9\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.776196 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.776029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37230471-1ae4-4696-b8c8-8c3c165dae31-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.778540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.778517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37230471-1ae4-4696-b8c8-8c3c165dae31-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.784215 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.784193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8g9\" (UniqueName: \"kubernetes.io/projected/37230471-1ae4-4696-b8c8-8c3c165dae31-kube-api-access-zn8g9\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") " pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.898745 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.898713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:48.950267 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:48.950121    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:00:49.174791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:49.174769    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"]
Sep 16 19:00:49.176478 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:00:49.176451    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37230471_1ae4_4696_b8c8_8c3c165dae31.slice/crio-83e43d61343306bab3072ed0467113d32d7feb6a96502d97b2a2b77b36003ec1 WatchSource:0}: Error finding container 83e43d61343306bab3072ed0467113d32d7feb6a96502d97b2a2b77b36003ec1: Status 404 returned error can't find the container with id 83e43d61343306bab3072ed0467113d32d7feb6a96502d97b2a2b77b36003ec1
Sep 16 19:00:49.510534 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:49.510443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" event={"ID":"37230471-1ae4-4696-b8c8-8c3c165dae31","Type":"ContainerStarted","Data":"1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689"}
Sep 16 19:00:49.510534 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:49.510505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" event={"ID":"37230471-1ae4-4696-b8c8-8c3c165dae31","Type":"ContainerStarted","Data":"83e43d61343306bab3072ed0467113d32d7feb6a96502d97b2a2b77b36003ec1"}
Sep 16 19:00:49.510709 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:49.510553    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:00:49.528297 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:49.528252    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" podStartSLOduration=1.528239559 podStartE2EDuration="1.528239559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:49.526770755 +0000 UTC m=+1033.638567429" watchObservedRunningTime="2026-09-16 19:00:49.528239559 +0000 UTC m=+1033.640036200"
Sep 16 19:00:53.041045 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:00:53.040992    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:20.517562 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:20.517530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:01:22.171778 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:22.171734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:22.204409 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:22.204372    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"]
Sep 16 19:01:22.248913 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:22.248861    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:01:30.517865 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:30.517782    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:37.224450 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:37.224401    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main" containerID="cri-o://1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689" gracePeriod=45
Sep 16 19:01:40.517760 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:40.517715    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:50.518144 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:50.518102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:01:50.518553 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:01:50.518202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:02:00.517406 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:00.517359    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:07.467504 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.467477    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:02:07.631913 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.631807    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn8g9\" (UniqueName: \"kubernetes.io/projected/37230471-1ae4-4696-b8c8-8c3c165dae31-kube-api-access-zn8g9\") pod \"37230471-1ae4-4696-b8c8-8c3c165dae31\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") "
Sep 16 19:02:07.632081 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.631934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37230471-1ae4-4696-b8c8-8c3c165dae31-tls-certs\") pod \"37230471-1ae4-4696-b8c8-8c3c165dae31\" (UID: \"37230471-1ae4-4696-b8c8-8c3c165dae31\") "
Sep 16 19:02:07.634123 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.634086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37230471-1ae4-4696-b8c8-8c3c165dae31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37230471-1ae4-4696-b8c8-8c3c165dae31" (UID: "37230471-1ae4-4696-b8c8-8c3c165dae31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:07.634240 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.634143    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37230471-1ae4-4696-b8c8-8c3c165dae31-kube-api-access-zn8g9" (OuterVolumeSpecName: "kube-api-access-zn8g9") pod "37230471-1ae4-4696-b8c8-8c3c165dae31" (UID: "37230471-1ae4-4696-b8c8-8c3c165dae31"). InnerVolumeSpecName "kube-api-access-zn8g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:07.732933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.732875    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8g9\" (UniqueName: \"kubernetes.io/projected/37230471-1ae4-4696-b8c8-8c3c165dae31-kube-api-access-zn8g9\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.732933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.732931    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37230471-1ae4-4696-b8c8-8c3c165dae31-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.911526 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.911489    2566 generic.go:356] "Generic (PLEG): container finished" podID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerID="1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689" exitCode=0
Sep 16 19:02:07.911526 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.911529    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" event={"ID":"37230471-1ae4-4696-b8c8-8c3c165dae31","Type":"ContainerDied","Data":"1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689"}
Sep 16 19:02:07.911819 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.911546    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"
Sep 16 19:02:07.911819 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.911600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l" event={"ID":"37230471-1ae4-4696-b8c8-8c3c165dae31","Type":"ContainerDied","Data":"83e43d61343306bab3072ed0467113d32d7feb6a96502d97b2a2b77b36003ec1"}
Sep 16 19:02:07.911819 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.911622    2566 scope.go:117] "RemoveContainer" containerID="1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689"
Sep 16 19:02:07.920285 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.920267    2566 scope.go:117] "RemoveContainer" containerID="1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689"
Sep 16 19:02:07.920550 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:02:07.920532    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689\": container with ID starting with 1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689 not found: ID does not exist" containerID="1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689"
Sep 16 19:02:07.920601 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.920557    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689"} err="failed to get container status \"1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689\": rpc error: code = NotFound desc = could not find container \"1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689\": container with ID starting with 1a10f40e906f98c80df7e5b404302dc9eaf897f4c5dbbeb676826c53f1df5689 not found: ID does not exist"
Sep 16 19:02:07.933076 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.933055    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"]
Sep 16 19:02:07.939263 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:07.939241    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-aae51b37/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-sche56l6l"]
Sep 16 19:02:08.403210 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:08.403178    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" path="/var/lib/kubelet/pods/37230471-1ae4-4696-b8c8-8c3c165dae31/volumes"
Sep 16 19:02:10.835755 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:10.835724    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"]
Sep 16 19:02:10.836712 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:10.836680    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:10.913082 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:10.913043    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:12.272379 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:12.272327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:12.371478 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:12.371433    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:18.306749 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:18.306710    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:19.194929 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:19.194866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:20.450330 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.450290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:20.686147 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.686110    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"]
Sep 16 19:02:20.686609 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.686588    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main"
Sep 16 19:02:20.686708 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.686615    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main"
Sep 16 19:02:20.686762 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.686719    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="37230471-1ae4-4696-b8c8-8c3c165dae31" containerName="main"
Sep 16 19:02:20.691520 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.691496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:20.698000 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.697971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"default-dockercfg-8tkjm\""
Sep 16 19:02:20.701850 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.698187    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"kube-root-ca.crt\""
Sep 16 19:02:20.701850 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.698032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-kserve-self-signed-certs\""
Sep 16 19:02:20.701850 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.698480    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"scheduler-configmap-ref-test-epp-sa-dockercfg-xtnbl\""
Sep 16 19:02:20.701850 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.698701    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-15d6126c\"/\"openshift-service-ca.crt\""
Sep 16 19:02:20.708343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.708316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"]
Sep 16 19:02:20.737687 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.737644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c82889-c797-42ac-86c1-89a4813f4efd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:20.737838 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.737709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkskm\" (UniqueName: \"kubernetes.io/projected/03c82889-c797-42ac-86c1-89a4813f4efd-kube-api-access-hkskm\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:20.767751 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.767712    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:20.838390 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.838347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c82889-c797-42ac-86c1-89a4813f4efd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:20.838390 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.838395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkskm\" (UniqueName: \"kubernetes.io/projected/03c82889-c797-42ac-86c1-89a4813f4efd-kube-api-access-hkskm\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:20.840936 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.840898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c82889-c797-42ac-86c1-89a4813f4efd-tls-certs\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:20.853505 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:20.853478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkskm\" (UniqueName: \"kubernetes.io/projected/03c82889-c797-42ac-86c1-89a4813f4efd-kube-api-access-hkskm\") pod \"scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") " pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:21.006887 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:21.006787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:21.286684 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:21.286654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"]
Sep 16 19:02:21.287670 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:02:21.287639    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c82889_c797_42ac_86c1_89a4813f4efd.slice/crio-179c5834b606154e8bc358a54a19c00a42dfc1b406cf7491ebbe303641e80251 WatchSource:0}: Error finding container 179c5834b606154e8bc358a54a19c00a42dfc1b406cf7491ebbe303641e80251: Status 404 returned error can't find the container with id 179c5834b606154e8bc358a54a19c00a42dfc1b406cf7491ebbe303641e80251
Sep 16 19:02:21.985046 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:21.985007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" event={"ID":"03c82889-c797-42ac-86c1-89a4813f4efd","Type":"ContainerStarted","Data":"399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067"}
Sep 16 19:02:21.985476 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:21.985060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" event={"ID":"03c82889-c797-42ac-86c1-89a4813f4efd","Type":"ContainerStarted","Data":"179c5834b606154e8bc358a54a19c00a42dfc1b406cf7491ebbe303641e80251"}
Sep 16 19:02:21.985476 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:21.985082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:22.008195 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:22.008147    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" podStartSLOduration=2.00812984 podStartE2EDuration="2.00812984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:22.006664126 +0000 UTC m=+1126.118460767" watchObservedRunningTime="2026-09-16 19:02:22.00812984 +0000 UTC m=+1126.119926480"
Sep 16 19:02:25.106437 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:25.106400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:25.170501 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:25.170437    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:25.856248 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:25.856211    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main" containerID="cri-o://f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a" gracePeriod=45
Sep 16 19:02:27.448637 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:27.448595    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:28.306875 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:28.306837    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:38.306396 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:38.306353    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:38.306810 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:38.306463    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:02:48.307248 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:48.307207    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:02:52.991689 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:52.991659    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:02:54.601555 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:54.601517    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"]
Sep 16 19:02:54.605427 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:54.605273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:54.686267 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:54.686231    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:02:56.106582 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.106559    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:02:56.156210 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.156180    2566 generic.go:356] "Generic (PLEG): container finished" podID="934c1f55-ba21-440d-a986-3300e64fff6d" containerID="f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a" exitCode=0
Sep 16 19:02:56.156391 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.156226    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"
Sep 16 19:02:56.156391 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.156265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" event={"ID":"934c1f55-ba21-440d-a986-3300e64fff6d","Type":"ContainerDied","Data":"f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a"}
Sep 16 19:02:56.156391 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.156333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq" event={"ID":"934c1f55-ba21-440d-a986-3300e64fff6d","Type":"ContainerDied","Data":"d5f697857d73be6e2f9021790d371e0536c8f24ed67d049adc568db6c5a2aad9"}
Sep 16 19:02:56.156391 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.156366    2566 scope.go:117] "RemoveContainer" containerID="f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a"
Sep 16 19:02:56.164372 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.164356    2566 scope.go:117] "RemoveContainer" containerID="f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a"
Sep 16 19:02:56.164629 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:02:56.164613    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a\": container with ID starting with f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a not found: ID does not exist" containerID="f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a"
Sep 16 19:02:56.164672 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.164637    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a"} err="failed to get container status \"f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a\": rpc error: code = NotFound desc = could not find container \"f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a\": container with ID starting with f197dfa6b8a2390988b9bc2404855a49ff3e0c2df0b7308d70cbe335e817d32a not found: ID does not exist"
Sep 16 19:02:56.241745 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.241715    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934c1f55-ba21-440d-a986-3300e64fff6d-tls-certs\") pod \"934c1f55-ba21-440d-a986-3300e64fff6d\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") "
Sep 16 19:02:56.241904 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.241768    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbkvq\" (UniqueName: \"kubernetes.io/projected/934c1f55-ba21-440d-a986-3300e64fff6d-kube-api-access-rbkvq\") pod \"934c1f55-ba21-440d-a986-3300e64fff6d\" (UID: \"934c1f55-ba21-440d-a986-3300e64fff6d\") "
Sep 16 19:02:56.243925 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.243895    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/934c1f55-ba21-440d-a986-3300e64fff6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "934c1f55-ba21-440d-a986-3300e64fff6d" (UID: "934c1f55-ba21-440d-a986-3300e64fff6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:56.244027 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.244001    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/934c1f55-ba21-440d-a986-3300e64fff6d-kube-api-access-rbkvq" (OuterVolumeSpecName: "kube-api-access-rbkvq") pod "934c1f55-ba21-440d-a986-3300e64fff6d" (UID: "934c1f55-ba21-440d-a986-3300e64fff6d"). InnerVolumeSpecName "kube-api-access-rbkvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:56.342671 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.342642    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/934c1f55-ba21-440d-a986-3300e64fff6d-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.342671 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.342667    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbkvq\" (UniqueName: \"kubernetes.io/projected/934c1f55-ba21-440d-a986-3300e64fff6d-kube-api-access-rbkvq\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:02:56.475244 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.475212    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"]
Sep 16 19:02:56.479705 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:56.479676    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-enabled-requires-token-d4ed2c08/auth-enabled-test-kserve-router-scheduler-547fb7d6bb-jkvwq"]
Sep 16 19:02:58.402303 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:02:58.402272    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" path="/var/lib/kubelet/pods/934c1f55-ba21-440d-a986-3300e64fff6d/volumes"
Sep 16 19:03:00.948946 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:00.948908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:01.055013 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:01.054978    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:02.992323 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:02.992280    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:07.365466 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:07.365421    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:08.951851 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:08.951807    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:09.206819 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:09.206730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:09.621198 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:09.621096    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main" containerID="cri-o://399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067" gracePeriod=45
Sep 16 19:03:11.905220 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:11.905186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:12.992482 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:12.992443    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:13.111976 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:13.111944    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:13.210711 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:13.210678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:18.473969 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:18.473920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:22.992228 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:22.992189    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:22.992663 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:22.992335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:03:32.171335 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:32.171294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:32.991620 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:32.991578    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:03:39.876583 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:39.876556    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:03:40.010202 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.010166    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkskm\" (UniqueName: \"kubernetes.io/projected/03c82889-c797-42ac-86c1-89a4813f4efd-kube-api-access-hkskm\") pod \"03c82889-c797-42ac-86c1-89a4813f4efd\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") "
Sep 16 19:03:40.010394 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.010301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c82889-c797-42ac-86c1-89a4813f4efd-tls-certs\") pod \"03c82889-c797-42ac-86c1-89a4813f4efd\" (UID: \"03c82889-c797-42ac-86c1-89a4813f4efd\") "
Sep 16 19:03:40.012393 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.012365    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c82889-c797-42ac-86c1-89a4813f4efd-kube-api-access-hkskm" (OuterVolumeSpecName: "kube-api-access-hkskm") pod "03c82889-c797-42ac-86c1-89a4813f4efd" (UID: "03c82889-c797-42ac-86c1-89a4813f4efd"). InnerVolumeSpecName "kube-api-access-hkskm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:40.012393 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.012368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03c82889-c797-42ac-86c1-89a4813f4efd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03c82889-c797-42ac-86c1-89a4813f4efd" (UID: "03c82889-c797-42ac-86c1-89a4813f4efd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:40.111611 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.111569    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03c82889-c797-42ac-86c1-89a4813f4efd-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.111611 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.111608    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkskm\" (UniqueName: \"kubernetes.io/projected/03c82889-c797-42ac-86c1-89a4813f4efd-kube-api-access-hkskm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:03:40.391140 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.391103    2566 generic.go:356] "Generic (PLEG): container finished" podID="03c82889-c797-42ac-86c1-89a4813f4efd" containerID="399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067" exitCode=0
Sep 16 19:03:40.391322 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.391145    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"
Sep 16 19:03:40.391322 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.391182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" event={"ID":"03c82889-c797-42ac-86c1-89a4813f4efd","Type":"ContainerDied","Data":"399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067"}
Sep 16 19:03:40.391322 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.391233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4" event={"ID":"03c82889-c797-42ac-86c1-89a4813f4efd","Type":"ContainerDied","Data":"179c5834b606154e8bc358a54a19c00a42dfc1b406cf7491ebbe303641e80251"}
Sep 16 19:03:40.391322 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.391256    2566 scope.go:117] "RemoveContainer" containerID="399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067"
Sep 16 19:03:40.403922 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.400550    2566 scope.go:117] "RemoveContainer" containerID="399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067"
Sep 16 19:03:40.403922 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:03:40.400986    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067\": container with ID starting with 399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067 not found: ID does not exist" containerID="399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067"
Sep 16 19:03:40.403922 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.401025    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067"} err="failed to get container status \"399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067\": rpc error: code = NotFound desc = could not find container \"399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067\": container with ID starting with 399270e899accb85a4de21b909b9dcacb284435a79977782c9911894f7baa067 not found: ID does not exist"
Sep 16 19:03:40.416291 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.416265    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"]
Sep 16 19:03:40.419782 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:40.419749    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-15d6126c/scheduler-configmap-ref-test-kserve-router-scheduler-566c9bqlf4"]
Sep 16 19:03:42.403505 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:42.403469    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" path="/var/lib/kubelet/pods/03c82889-c797-42ac-86c1-89a4813f4efd/volumes"
Sep 16 19:03:43.971306 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:43.971247    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:44.065457 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:44.065413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:50.617065 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:50.617027    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:52.360794 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.360756    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:52.577085 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577053    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"]
Sep 16 19:03:52.577482 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577455    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main"
Sep 16 19:03:52.577482 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577481    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main"
Sep 16 19:03:52.577713 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577502    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main"
Sep 16 19:03:52.577713 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577511    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main"
Sep 16 19:03:52.577713 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577615    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c82889-c797-42ac-86c1-89a4813f4efd" containerName="main"
Sep 16 19:03:52.577713 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.577634    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="934c1f55-ba21-440d-a986-3300e64fff6d" containerName="main"
Sep 16 19:03:52.582591 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.582121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.586523 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.586496    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"openshift-service-ca.crt\""
Sep 16 19:03:52.586650 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.586527    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Sep 16 19:03:52.586650 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.586563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"kube-root-ca.crt\""
Sep 16 19:03:52.586895 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.586856    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-rcjxx\""
Sep 16 19:03:52.587020 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.586924    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-2541cd65\"/\"default-dockercfg-hr6fv\""
Sep 16 19:03:52.592262 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.592236    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"]
Sep 16 19:03:52.633268 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.633181    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:52.712722 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.712691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5309498-cc93-4ef0-bbe3-497bcfa6767c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.712863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.712729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpjd8\" (UniqueName: \"kubernetes.io/projected/b5309498-cc93-4ef0-bbe3-497bcfa6767c-kube-api-access-mpjd8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.814199 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.814167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5309498-cc93-4ef0-bbe3-497bcfa6767c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.814365 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.814206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpjd8\" (UniqueName: \"kubernetes.io/projected/b5309498-cc93-4ef0-bbe3-497bcfa6767c-kube-api-access-mpjd8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.816748 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.816728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5309498-cc93-4ef0-bbe3-497bcfa6767c-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.824257 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.824233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpjd8\" (UniqueName: \"kubernetes.io/projected/b5309498-cc93-4ef0-bbe3-497bcfa6767c-kube-api-access-mpjd8\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:52.896043 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:52.896007    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:53.176952 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:53.176918    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"]
Sep 16 19:03:53.177967 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:03:53.177946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5309498_cc93_4ef0_bbe3_497bcfa6767c.slice/crio-8d6ac75d366d161018d1869dfd624a18c9f737caa83bcfd2b80c1a9a6f03205b WatchSource:0}: Error finding container 8d6ac75d366d161018d1869dfd624a18c9f737caa83bcfd2b80c1a9a6f03205b: Status 404 returned error can't find the container with id 8d6ac75d366d161018d1869dfd624a18c9f737caa83bcfd2b80c1a9a6f03205b
Sep 16 19:03:53.460519 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:53.460418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" event={"ID":"b5309498-cc93-4ef0-bbe3-497bcfa6767c","Type":"ContainerStarted","Data":"6f705a13c572b6ddd2b84f4cbc44f07f43ad917e2d9950e1b5c0e08915820483"}
Sep 16 19:03:53.460519 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:53.460480    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:03:53.460519 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:53.460502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" event={"ID":"b5309498-cc93-4ef0-bbe3-497bcfa6767c","Type":"ContainerStarted","Data":"8d6ac75d366d161018d1869dfd624a18c9f737caa83bcfd2b80c1a9a6f03205b"}
Sep 16 19:03:53.480521 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:53.480471    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" podStartSLOduration=1.4804558939999999 podStartE2EDuration="1.480455894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:53.478849968 +0000 UTC m=+1217.590646611" watchObservedRunningTime="2026-09-16 19:03:53.480455894 +0000 UTC m=+1217.592252534"
Sep 16 19:03:56.836923 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:56.836891    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:03:56.949487 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:03:56.949447    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:06.408730 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:06.408693    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:24.468268 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:24.468236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:04:25.963218 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:25.963159    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:25.991637 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:25.991600    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"]
Sep 16 19:04:27.159854 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:27.159815    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:34.468145 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:34.468106    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:04:35.084639 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:35.084602    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:04:37.477382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:37.476940    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" message="" err="command '/bin/sleep 15' exited with 137: "
Sep 16 19:04:37.477382 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:04:37.476985    2566 kuberuntime_container.go:755] "PreStop hook failed" err="command '/bin/sleep 15' exited with 137: " pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerName="main" containerID="cri-o://6f705a13c572b6ddd2b84f4cbc44f07f43ad917e2d9950e1b5c0e08915820483"
Sep 16 19:04:37.477382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:37.477069    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerName="main" containerID="cri-o://6f705a13c572b6ddd2b84f4cbc44f07f43ad917e2d9950e1b5c0e08915820483" gracePeriod=49
Sep 16 19:04:37.684450 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:37.684422    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc_b5309498-cc93-4ef0-bbe3-497bcfa6767c/main/0.log"
Sep 16 19:04:37.684625 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:37.684462    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerID="6f705a13c572b6ddd2b84f4cbc44f07f43ad917e2d9950e1b5c0e08915820483" exitCode=1
Sep 16 19:04:37.684625 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:37.684501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" event={"ID":"b5309498-cc93-4ef0-bbe3-497bcfa6767c","Type":"ContainerDied","Data":"6f705a13c572b6ddd2b84f4cbc44f07f43ad917e2d9950e1b5c0e08915820483"}
Sep 16 19:04:38.107780 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.107758    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc_b5309498-cc93-4ef0-bbe3-497bcfa6767c/main/0.log"
Sep 16 19:04:38.110010 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.109993    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:04:38.203784 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.203755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpjd8\" (UniqueName: \"kubernetes.io/projected/b5309498-cc93-4ef0-bbe3-497bcfa6767c-kube-api-access-mpjd8\") pod \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") "
Sep 16 19:04:38.203784 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.203791    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5309498-cc93-4ef0-bbe3-497bcfa6767c-tls-certs\") pod \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\" (UID: \"b5309498-cc93-4ef0-bbe3-497bcfa6767c\") "
Sep 16 19:04:38.206079 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.206047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5309498-cc93-4ef0-bbe3-497bcfa6767c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5309498-cc93-4ef0-bbe3-497bcfa6767c" (UID: "b5309498-cc93-4ef0-bbe3-497bcfa6767c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:38.206201 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.206177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5309498-cc93-4ef0-bbe3-497bcfa6767c-kube-api-access-mpjd8" (OuterVolumeSpecName: "kube-api-access-mpjd8") pod "b5309498-cc93-4ef0-bbe3-497bcfa6767c" (UID: "b5309498-cc93-4ef0-bbe3-497bcfa6767c"). InnerVolumeSpecName "kube-api-access-mpjd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:38.305168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.305096    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpjd8\" (UniqueName: \"kubernetes.io/projected/b5309498-cc93-4ef0-bbe3-497bcfa6767c-kube-api-access-mpjd8\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.305168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.305119    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5309498-cc93-4ef0-bbe3-497bcfa6767c-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:04:38.692064 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.692033    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-2541cd65_scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc_b5309498-cc93-4ef0-bbe3-497bcfa6767c/main/0.log"
Sep 16 19:04:38.692565 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.692090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc" event={"ID":"b5309498-cc93-4ef0-bbe3-497bcfa6767c","Type":"ContainerDied","Data":"8d6ac75d366d161018d1869dfd624a18c9f737caa83bcfd2b80c1a9a6f03205b"}
Sep 16 19:04:38.692565 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.692144    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"
Sep 16 19:04:38.692565 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.692154    2566 scope.go:117] "RemoveContainer" containerID="6f705a13c572b6ddd2b84f4cbc44f07f43ad917e2d9950e1b5c0e08915820483"
Sep 16 19:04:38.715307 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.715274    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"]
Sep 16 19:04:38.722874 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:38.722846    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-2541cd65/scheduler-ha-replicas-test-kserve-router-scheduler-557bbc5lqkzc"]
Sep 16 19:04:40.402366 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:04:40.402329    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" path="/var/lib/kubelet/pods/b5309498-cc93-4ef0-bbe3-497bcfa6767c/volumes"
Sep 16 19:05:15.528875 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:15.528834    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:15.718302 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:15.718249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:22.520609 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:22.520562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:23.702337 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.702294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:23.876830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.876797    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"]
Sep 16 19:05:23.877177 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.877162    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerName="main"
Sep 16 19:05:23.877256 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.877181    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerName="main"
Sep 16 19:05:23.877304 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.877257    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5309498-cc93-4ef0-bbe3-497bcfa6767c" containerName="main"
Sep 16 19:05:23.880483 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.880461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:23.884360 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.884332    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-epp-sa-dockercfg-hvrp9\""
Sep 16 19:05:23.884360 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.884336    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"openshift-service-ca.crt\""
Sep 16 19:05:23.884546 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.884335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"kube-root-ca.crt\""
Sep 16 19:05:23.884546 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.884399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"default-dockercfg-vj2jl\""
Sep 16 19:05:23.884546 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.884399    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-35377f35\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Sep 16 19:05:23.891693 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.891670    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"]
Sep 16 19:05:23.903318 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.903284    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:23.979612 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.979531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:23.979612 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.979567    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:23.979806 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.979651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8d2r\" (UniqueName: \"kubernetes.io/projected/27721a0b-1ace-4873-9f2a-bf04bb39dee7-kube-api-access-g8d2r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:23.979806 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:23.979754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.081106 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.081064    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.081285 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.081115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.081285 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.081149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.081400 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.081280    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8d2r\" (UniqueName: \"kubernetes.io/projected/27721a0b-1ace-4873-9f2a-bf04bb39dee7-kube-api-access-g8d2r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.081587 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.081566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tokenizer-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.081644 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.081588    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-cachi2-cache\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.083825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.083795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tls-certs\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.090309 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.090287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8d2r\" (UniqueName: \"kubernetes.io/projected/27721a0b-1ace-4873-9f2a-bf04bb39dee7-kube-api-access-g8d2r\") pod \"scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") " pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.191654 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.191619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.468952 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.468924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"]
Sep 16 19:05:24.470285 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:05:24.470258    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27721a0b_1ace_4873_9f2a_bf04bb39dee7.slice/crio-2acf95d061da68a8b5d60570cb684e1cf56f37577405d64b866a012485727d48 WatchSource:0}: Error finding container 2acf95d061da68a8b5d60570cb684e1cf56f37577405d64b866a012485727d48: Status 404 returned error can't find the container with id 2acf95d061da68a8b5d60570cb684e1cf56f37577405d64b866a012485727d48
Sep 16 19:05:24.924725 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.924692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" event={"ID":"27721a0b-1ace-4873-9f2a-bf04bb39dee7","Type":"ContainerStarted","Data":"0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c"}
Sep 16 19:05:24.925224 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.924750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" event={"ID":"27721a0b-1ace-4873-9f2a-bf04bb39dee7","Type":"ContainerStarted","Data":"2acf95d061da68a8b5d60570cb684e1cf56f37577405d64b866a012485727d48"}
Sep 16 19:05:24.925224 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.924835    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:24.948030 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:24.947979    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" podStartSLOduration=1.9479609679999998 podStartE2EDuration="1.947960968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:24.946256675 +0000 UTC m=+1309.058053317" watchObservedRunningTime="2026-09-16 19:05:24.947960968 +0000 UTC m=+1309.059757610"
Sep 16 19:05:29.066173 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:29.066136    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:29.180441 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:29.180400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:34.474383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:34.474347    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:38.590988 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:38.590944    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:38.682381 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:38.682345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:55.934386 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:55.934353    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:05:57.898707 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:57.898655    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:57.932403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:57.932367    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"]
Sep 16 19:05:58.039378 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:58.039338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:58.983990 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:58.983872    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:05:59.084283 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:05:59.084248    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:05.934540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:05.934500    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:12.952004 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:12.951933    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main" containerID="cri-o://0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c" gracePeriod=45
Sep 16 19:06:15.934542 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:15.934503    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:25.935245 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:25.935202    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:25.935685 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:25.935313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:06:27.810818 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:27.810780    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:27.913793 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:27.913758    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:28.016246 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:28.016213    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:34.540424 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:34.540384    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:35.934874 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:35.934834    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:06:40.657154 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:40.657117    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:40.754066 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:40.754025    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:43.206368 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.206343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:06:43.330864 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.330765    2566 generic.go:356] "Generic (PLEG): container finished" podID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerID="0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c" exitCode=0
Sep 16 19:06:43.330864 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.330809    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"
Sep 16 19:06:43.330864 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.330844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" event={"ID":"27721a0b-1ace-4873-9f2a-bf04bb39dee7","Type":"ContainerDied","Data":"0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c"}
Sep 16 19:06:43.331140 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.330935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk" event={"ID":"27721a0b-1ace-4873-9f2a-bf04bb39dee7","Type":"ContainerDied","Data":"2acf95d061da68a8b5d60570cb684e1cf56f37577405d64b866a012485727d48"}
Sep 16 19:06:43.331140 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.330962    2566 scope.go:117] "RemoveContainer" containerID="0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c"
Sep 16 19:06:43.340572 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.340554    2566 scope.go:117] "RemoveContainer" containerID="0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c"
Sep 16 19:06:43.340835 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:06:43.340807    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c\": container with ID starting with 0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c not found: ID does not exist" containerID="0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c"
Sep 16 19:06:43.340962 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.340843    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c"} err="failed to get container status \"0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c\": rpc error: code = NotFound desc = could not find container \"0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c\": container with ID starting with 0120a644e25b71648220552ef0eced4f95d93e727f0560b5a292a8224dcdda8c not found: ID does not exist"
Sep 16 19:06:43.359229 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tokenizer-cache\") pod \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") "
Sep 16 19:06:43.359310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359249    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-cachi2-cache\") pod \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") "
Sep 16 19:06:43.359310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tls-certs\") pod \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") "
Sep 16 19:06:43.359386 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359324    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8d2r\" (UniqueName: \"kubernetes.io/projected/27721a0b-1ace-4873-9f2a-bf04bb39dee7-kube-api-access-g8d2r\") pod \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\" (UID: \"27721a0b-1ace-4873-9f2a-bf04bb39dee7\") "
Sep 16 19:06:43.359517 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359476    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tokenizer-cache" (OuterVolumeSpecName: "tokenizer-cache") pod "27721a0b-1ace-4873-9f2a-bf04bb39dee7" (UID: "27721a0b-1ace-4873-9f2a-bf04bb39dee7"). InnerVolumeSpecName "tokenizer-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:43.359590 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-cachi2-cache" (OuterVolumeSpecName: "cachi2-cache") pod "27721a0b-1ace-4873-9f2a-bf04bb39dee7" (UID: "27721a0b-1ace-4873-9f2a-bf04bb39dee7"). InnerVolumeSpecName "cachi2-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:43.359652 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359614    2566 reconciler_common.go:299] "Volume detached for volume \"tokenizer-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tokenizer-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.359652 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.359627    2566 reconciler_common.go:299] "Volume detached for volume \"cachi2-cache\" (UniqueName: \"kubernetes.io/empty-dir/27721a0b-1ace-4873-9f2a-bf04bb39dee7-cachi2-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.361500 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.361471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27721a0b-1ace-4873-9f2a-bf04bb39dee7" (UID: "27721a0b-1ace-4873-9f2a-bf04bb39dee7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:43.361500 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.361492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27721a0b-1ace-4873-9f2a-bf04bb39dee7-kube-api-access-g8d2r" (OuterVolumeSpecName: "kube-api-access-g8d2r") pod "27721a0b-1ace-4873-9f2a-bf04bb39dee7" (UID: "27721a0b-1ace-4873-9f2a-bf04bb39dee7"). InnerVolumeSpecName "kube-api-access-g8d2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:43.461044 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.461004    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8d2r\" (UniqueName: \"kubernetes.io/projected/27721a0b-1ace-4873-9f2a-bf04bb39dee7-kube-api-access-g8d2r\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.461044 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.461039    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27721a0b-1ace-4873-9f2a-bf04bb39dee7-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:06:43.654436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.654388    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"]
Sep 16 19:06:43.656868 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:43.656844    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-35377f35/scheduler-custom-template-test-kserve-router-scheduler-ffcq45qk"]
Sep 16 19:06:44.403324 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:44.403291    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" path="/var/lib/kubelet/pods/27721a0b-1ace-4873-9f2a-bf04bb39dee7/volumes"
Sep 16 19:06:47.772810 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:47.772779    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:47.866565 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:47.866529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:54.469334 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:54.469290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:57.896496 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:57.896456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:58.125476 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.125443    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"]
Sep 16 19:06:58.125791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.125778    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main"
Sep 16 19:06:58.125859 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.125792    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main"
Sep 16 19:06:58.125922 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.125862    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="27721a0b-1ace-4873-9f2a-bf04bb39dee7" containerName="main"
Sep 16 19:06:58.129725 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.129693    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.132476 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.132446    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"kube-root-ca.crt\""
Sep 16 19:06:58.133581 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.133561    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-kserve-self-signed-certs\""
Sep 16 19:06:58.133728 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.133702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"default-dockercfg-sv6vn\""
Sep 16 19:06:58.133854 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.133832    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"openshift-service-ca.crt\""
Sep 16 19:06:58.133931 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.133857    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-auth-disabled-no-token-required-396e4821\"/\"auth-disabled-test-epp-sa-dockercfg-b2tkx\""
Sep 16 19:06:58.140203 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.140172    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"]
Sep 16 19:06:58.163433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.163357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:06:58.173392 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.173365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm6tf\" (UniqueName: \"kubernetes.io/projected/3b338e3c-228b-4b01-96d0-6ecc71461193-kube-api-access-mm6tf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.173544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.173415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b338e3c-228b-4b01-96d0-6ecc71461193-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.274392 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.274358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b338e3c-228b-4b01-96d0-6ecc71461193-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.274585 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.274428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm6tf\" (UniqueName: \"kubernetes.io/projected/3b338e3c-228b-4b01-96d0-6ecc71461193-kube-api-access-mm6tf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.276842 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.276813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b338e3c-228b-4b01-96d0-6ecc71461193-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.287317 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.287292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm6tf\" (UniqueName: \"kubernetes.io/projected/3b338e3c-228b-4b01-96d0-6ecc71461193-kube-api-access-mm6tf\") pod \"auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") " pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.440984 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.440863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:06:58.728910 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:58.728832    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"]
Sep 16 19:06:58.731562 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:06:58.731529    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b338e3c_228b_4b01_96d0_6ecc71461193.slice/crio-7401b43c5dca5cb42c4da8bfb80389f32edaf5cbf4437259663e78e9297857ac WatchSource:0}: Error finding container 7401b43c5dca5cb42c4da8bfb80389f32edaf5cbf4437259663e78e9297857ac: Status 404 returned error can't find the container with id 7401b43c5dca5cb42c4da8bfb80389f32edaf5cbf4437259663e78e9297857ac
Sep 16 19:06:59.418795 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:59.418759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" event={"ID":"3b338e3c-228b-4b01-96d0-6ecc71461193","Type":"ContainerStarted","Data":"56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff"}
Sep 16 19:06:59.419203 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:59.418808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" event={"ID":"3b338e3c-228b-4b01-96d0-6ecc71461193","Type":"ContainerStarted","Data":"7401b43c5dca5cb42c4da8bfb80389f32edaf5cbf4437259663e78e9297857ac"}
Sep 16 19:06:59.419203 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:06:59.418904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:07:01.425497 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:01.425456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:01.589383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:01.589321    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:10.396856 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:10.396797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:25.228418 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:25.228377    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:25.308721 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:25.308680    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:25.512203 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:25.512101    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:07:30.425747 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:30.425648    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:07:30.448343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:30.448294    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" podStartSLOduration=32.448277932 podStartE2EDuration="32.448277932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:59.44120619 +0000 UTC m=+1403.553002832" watchObservedRunningTime="2026-09-16 19:07:30.448277932 +0000 UTC m=+1434.560074572"
Sep 16 19:07:47.990103 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:07:47.990063    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:52.421744 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:08:52.421698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:08:52.499828 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:08:52.499784    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"]
Sep 16 19:08:52.570830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:08:52.570796    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:00.426352 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:00.426270    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:05.410181 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:05.410147    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:05.501012 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:05.500973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:07.518997 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:07.518949    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main" containerID="cri-o://56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff" gracePeriod=45
Sep 16 19:09:10.426087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:10.426047    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:20.425989 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:20.425952    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:20.426354 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:20.426058    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:09:22.581069 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:22.581033    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:30.425778 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:30.425736    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:09:37.775149 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.775120    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:09:37.820270 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.820188    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b338e3c-228b-4b01-96d0-6ecc71461193-tls-certs\") pod \"3b338e3c-228b-4b01-96d0-6ecc71461193\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") "
Sep 16 19:09:37.820410 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.820304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm6tf\" (UniqueName: \"kubernetes.io/projected/3b338e3c-228b-4b01-96d0-6ecc71461193-kube-api-access-mm6tf\") pod \"3b338e3c-228b-4b01-96d0-6ecc71461193\" (UID: \"3b338e3c-228b-4b01-96d0-6ecc71461193\") "
Sep 16 19:09:37.822492 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.822455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b338e3c-228b-4b01-96d0-6ecc71461193-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3b338e3c-228b-4b01-96d0-6ecc71461193" (UID: "3b338e3c-228b-4b01-96d0-6ecc71461193"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:37.822604 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.822514    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b338e3c-228b-4b01-96d0-6ecc71461193-kube-api-access-mm6tf" (OuterVolumeSpecName: "kube-api-access-mm6tf") pod "3b338e3c-228b-4b01-96d0-6ecc71461193" (UID: "3b338e3c-228b-4b01-96d0-6ecc71461193"). InnerVolumeSpecName "kube-api-access-mm6tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:37.921340 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.921305    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b338e3c-228b-4b01-96d0-6ecc71461193-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:37.921340 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:37.921332    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm6tf\" (UniqueName: \"kubernetes.io/projected/3b338e3c-228b-4b01-96d0-6ecc71461193-kube-api-access-mm6tf\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:09:38.241318 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.241282    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerID="56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff" exitCode=0
Sep 16 19:09:38.241318 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.241320    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"
Sep 16 19:09:38.241535 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.241365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" event={"ID":"3b338e3c-228b-4b01-96d0-6ecc71461193","Type":"ContainerDied","Data":"56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff"}
Sep 16 19:09:38.241535 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.241416    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786" event={"ID":"3b338e3c-228b-4b01-96d0-6ecc71461193","Type":"ContainerDied","Data":"7401b43c5dca5cb42c4da8bfb80389f32edaf5cbf4437259663e78e9297857ac"}
Sep 16 19:09:38.241535 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.241443    2566 scope.go:117] "RemoveContainer" containerID="56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff"
Sep 16 19:09:38.251060 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.251003    2566 scope.go:117] "RemoveContainer" containerID="56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff"
Sep 16 19:09:38.251295 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:09:38.251278    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff\": container with ID starting with 56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff not found: ID does not exist" containerID="56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff"
Sep 16 19:09:38.251355 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.251302    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff"} err="failed to get container status \"56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff\": rpc error: code = NotFound desc = could not find container \"56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff\": container with ID starting with 56b6b6281991a4bdcee463ac0acf2a98c556e20cc06ca841d71759447c11deff not found: ID does not exist"
Sep 16 19:09:38.265899 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.265844    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"]
Sep 16 19:09:38.272029 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.271996    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-auth-disabled-no-token-required-396e4821/auth-disabled-test-kserve-router-scheduler-596d9bb85f-5d786"]
Sep 16 19:09:38.402561 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:38.402514    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" path="/var/lib/kubelet/pods/3b338e3c-228b-4b01-96d0-6ecc71461193/volumes"
Sep 16 19:09:42.478680 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:42.478641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:44.234696 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:44.234643    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:48.839203 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:48.839162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:52.638739 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.638701    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"]
Sep 16 19:09:52.639164 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.639051    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main"
Sep 16 19:09:52.639164 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.639064    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main"
Sep 16 19:09:52.639164 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.639124    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b338e3c-228b-4b01-96d0-6ecc71461193" containerName="main"
Sep 16 19:09:52.643401 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.643382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.647358 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.647336    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"default-dockercfg-zhwz6\""
Sep 16 19:09:52.648357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.648335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"openshift-service-ca.crt\""
Sep 16 19:09:52.648473 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.648368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"canary-v1-kserve-self-signed-certs\""
Sep 16 19:09:52.648473 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.648374    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-canary-service-backend-7b63470c\"/\"kube-root-ca.crt\""
Sep 16 19:09:52.668858 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.668827    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"]
Sep 16 19:09:52.695139 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.695108    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:52.745358 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.745319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db5cb59-0d2d-48a0-92c2-e0e53e854108-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.745547 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.745364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-dshm\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.745547 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.745401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-model-cache\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.745547 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.745471    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-home\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.745547 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.745515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.745706 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.745560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg88b\" (UniqueName: \"kubernetes.io/projected/4db5cb59-0d2d-48a0-92c2-e0e53e854108-kube-api-access-mg88b\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.846704 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.846670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg88b\" (UniqueName: \"kubernetes.io/projected/4db5cb59-0d2d-48a0-92c2-e0e53e854108-kube-api-access-mg88b\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.846874 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.846714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db5cb59-0d2d-48a0-92c2-e0e53e854108-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.846874 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.846738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-dshm\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.846874 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.846852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-model-cache\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.847041 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.846967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-home\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.847041 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.847015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.847268 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.847236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-model-cache\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.847394 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.847244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-home\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.847394 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.847296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-tmp-dir\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.849071 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.849049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4db5cb59-0d2d-48a0-92c2-e0e53e854108-dshm\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.849347 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.849329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4db5cb59-0d2d-48a0-92c2-e0e53e854108-tls-certs\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.857168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.857142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg88b\" (UniqueName: \"kubernetes.io/projected/4db5cb59-0d2d-48a0-92c2-e0e53e854108-kube-api-access-mg88b\") pod \"canary-v1-kserve-65d58ccc9-2b5js\" (UID: \"4db5cb59-0d2d-48a0-92c2-e0e53e854108\") " pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:52.953593 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:52.953512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:53.230556 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:53.230529    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"]
Sep 16 19:09:53.231976 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:09:53.231946    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db5cb59_0d2d_48a0_92c2_e0e53e854108.slice/crio-35610ee95f6b597008e2fdf8ab2053d81fd9311061bbd070b5b833c0dea65dd2 WatchSource:0}: Error finding container 35610ee95f6b597008e2fdf8ab2053d81fd9311061bbd070b5b833c0dea65dd2: Status 404 returned error can't find the container with id 35610ee95f6b597008e2fdf8ab2053d81fd9311061bbd070b5b833c0dea65dd2
Sep 16 19:09:53.233857 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:53.233835    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:09:53.321218 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:53.321186    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js" event={"ID":"4db5cb59-0d2d-48a0-92c2-e0e53e854108","Type":"ContainerStarted","Data":"35610ee95f6b597008e2fdf8ab2053d81fd9311061bbd070b5b833c0dea65dd2"}
Sep 16 19:09:54.328803 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:54.328766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js" event={"ID":"4db5cb59-0d2d-48a0-92c2-e0e53e854108","Type":"ContainerStarted","Data":"606d0f0a1d33f56321aec9b60ac3757cd3cd9533b03b4ebdb4c167cd4854e901"}
Sep 16 19:09:54.361674 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:54.361620    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js" podStartSLOduration=1.383075236 podStartE2EDuration="2.361604849s" totalImagesPullingTime="978.529613ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:54.358645256 +0000 UTC m=+1578.470441896" watchObservedRunningTime="2026-09-16 19:09:54.361604849 +0000 UTC m=+1578.473401490"
Sep 16 19:09:54.954668 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:54.954631    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:09:54.974675 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:54.974638    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.076529 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:55.076499    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.283066 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:55.282970    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:09:55.377368 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:09:55.377331    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:02.054861 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:02.054819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:02.954435 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:02.954395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:10:02.967421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:02.967391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:10:03.387586 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:03.387555    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-canary-service-backend-7b63470c/canary-v1-kserve-65d58ccc9-2b5js"
Sep 16 19:10:05.015089 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:05.015056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:09.295966 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:09.295933    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:09.361792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:09.361756    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:16.203380 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:16.203342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:19.428544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:19.428510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:31.085642 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.083135    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"]
Sep 16 19:10:31.089220 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.089194    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.093422 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.093397    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"kube-root-ca.crt\""
Sep 16 19:10:31.093560 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.093395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"default-dockercfg-ldrjs\""
Sep 16 19:10:31.093560 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.093395    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"openshift-service-ca.crt\""
Sep 16 19:10:31.093560 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.093456    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-40b85f95\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Sep 16 19:10:31.108974 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.108950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"]
Sep 16 19:10:31.147808 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.147782    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:31.166075 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.166048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.166177 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.166094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc49762-8757-4926-8999-43004f0641c9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.166177 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.166121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clx8w\" (UniqueName: \"kubernetes.io/projected/1cc49762-8757-4926-8999-43004f0641c9-kube-api-access-clx8w\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.166177 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.166161    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.166283 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.166191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.166283 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.166215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267152 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267279 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267279 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267279 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc49762-8757-4926-8999-43004f0641c9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267437 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clx8w\" (UniqueName: \"kubernetes.io/projected/1cc49762-8757-4926-8999-43004f0641c9-kube-api-access-clx8w\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267437 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267334    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267585 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267585 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267575    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.267703 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.267666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-home\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.269506 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.269484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.269660 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.269642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc49762-8757-4926-8999-43004f0641c9-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.279997 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.279970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clx8w\" (UniqueName: \"kubernetes.io/projected/1cc49762-8757-4926-8999-43004f0641c9-kube-api-access-clx8w\") pod \"scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") " pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.391870 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.391837    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:10:31.400063 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.400038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:31.672039 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:31.672016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"]
Sep 16 19:10:31.674126 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:10:31.674092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc49762_8757_4926_8999_43004f0641c9.slice/crio-a73b0ef20847c359a2d593fa67be1ec37fb9e9a6504eda52c9bd2b357f28728d WatchSource:0}: Error finding container a73b0ef20847c359a2d593fa67be1ec37fb9e9a6504eda52c9bd2b357f28728d: Status 404 returned error can't find the container with id a73b0ef20847c359a2d593fa67be1ec37fb9e9a6504eda52c9bd2b357f28728d
Sep 16 19:10:32.525924 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:32.525858    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" event={"ID":"1cc49762-8757-4926-8999-43004f0641c9","Type":"ContainerStarted","Data":"a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c"}
Sep 16 19:10:32.526261 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:32.525947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" event={"ID":"1cc49762-8757-4926-8999-43004f0641c9","Type":"ContainerStarted","Data":"a73b0ef20847c359a2d593fa67be1ec37fb9e9a6504eda52c9bd2b357f28728d"}
Sep 16 19:10:32.547202 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:32.547142    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" podStartSLOduration=1.547122074 podStartE2EDuration="1.547122074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:32.544560421 +0000 UTC m=+1616.656357111" watchObservedRunningTime="2026-09-16 19:10:32.547122074 +0000 UTC m=+1616.658918714"
Sep 16 19:10:33.401097 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:33.401062    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:41.400678 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:41.400641    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:41.412996 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:41.412971    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:10:41.586093 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:10:41.586060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:11:18.014210 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:18.014169    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:25.506869 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.506829    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:25.511428 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.511395    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"]
Sep 16 19:11:25.520356 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:11:25.520322    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:11:25.520356 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:25.520356 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:25.520356 ip-10-0-143-190 kubenswrapper[2566]:  > containerID="a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c" cmd=["/bin/sleep","15"]
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.520378    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" message="" err=<
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]:  >
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:11:25.520396    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" podUID="1cc49762-8757-4926-8999-43004f0641c9" containerName="main" containerID="cri-o://a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c"
Sep 16 19:11:25.520578 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.520443    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" podUID="1cc49762-8757-4926-8999-43004f0641c9" containerName="main" containerID="cri-o://a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c" gracePeriod=60
Sep 16 19:11:25.575727 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.575680    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" podUID="1cc49762-8757-4926-8999-43004f0641c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Sep 16 19:11:25.579331 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.579304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:11:25.803787 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.803753    2566 generic.go:356] "Generic (PLEG): container finished" podID="1cc49762-8757-4926-8999-43004f0641c9" containerID="a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c" exitCode=0
Sep 16 19:11:25.803975 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.803832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" event={"ID":"1cc49762-8757-4926-8999-43004f0641c9","Type":"ContainerDied","Data":"a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c"}
Sep 16 19:11:25.874006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:25.873969    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:11:26.024073 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc49762-8757-4926-8999-43004f0641c9-tls-certs\") pod \"1cc49762-8757-4926-8999-43004f0641c9\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") "
Sep 16 19:11:26.024073 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024078    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-model-cache\") pod \"1cc49762-8757-4926-8999-43004f0641c9\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") "
Sep 16 19:11:26.024326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-tmp-dir\") pod \"1cc49762-8757-4926-8999-43004f0641c9\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") "
Sep 16 19:11:26.024326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-dshm\") pod \"1cc49762-8757-4926-8999-43004f0641c9\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") "
Sep 16 19:11:26.024326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024238    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clx8w\" (UniqueName: \"kubernetes.io/projected/1cc49762-8757-4926-8999-43004f0641c9-kube-api-access-clx8w\") pod \"1cc49762-8757-4926-8999-43004f0641c9\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") "
Sep 16 19:11:26.024326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-home\") pod \"1cc49762-8757-4926-8999-43004f0641c9\" (UID: \"1cc49762-8757-4926-8999-43004f0641c9\") "
Sep 16 19:11:26.024526 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024386    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1cc49762-8757-4926-8999-43004f0641c9" (UID: "1cc49762-8757-4926-8999-43004f0641c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.024632 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024606    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.024765 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024622    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-home" (OuterVolumeSpecName: "home") pod "1cc49762-8757-4926-8999-43004f0641c9" (UID: "1cc49762-8757-4926-8999-43004f0641c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.024765 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.024683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-model-cache" (OuterVolumeSpecName: "model-cache") pod "1cc49762-8757-4926-8999-43004f0641c9" (UID: "1cc49762-8757-4926-8999-43004f0641c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.026432 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.026403    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-dshm" (OuterVolumeSpecName: "dshm") pod "1cc49762-8757-4926-8999-43004f0641c9" (UID: "1cc49762-8757-4926-8999-43004f0641c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:11:26.026528 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.026462    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc49762-8757-4926-8999-43004f0641c9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1cc49762-8757-4926-8999-43004f0641c9" (UID: "1cc49762-8757-4926-8999-43004f0641c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:11:26.026614 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.026596    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc49762-8757-4926-8999-43004f0641c9-kube-api-access-clx8w" (OuterVolumeSpecName: "kube-api-access-clx8w") pod "1cc49762-8757-4926-8999-43004f0641c9" (UID: "1cc49762-8757-4926-8999-43004f0641c9"). InnerVolumeSpecName "kube-api-access-clx8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:11:26.125357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.125256    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc49762-8757-4926-8999-43004f0641c9-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.125357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.125287    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.125357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.125296    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.125357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.125304    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clx8w\" (UniqueName: \"kubernetes.io/projected/1cc49762-8757-4926-8999-43004f0641c9-kube-api-access-clx8w\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.125357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.125330    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc49762-8757-4926-8999-43004f0641c9-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:11:26.810015 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.809938    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"
Sep 16 19:11:26.810415 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.809938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl" event={"ID":"1cc49762-8757-4926-8999-43004f0641c9","Type":"ContainerDied","Data":"a73b0ef20847c359a2d593fa67be1ec37fb9e9a6504eda52c9bd2b357f28728d"}
Sep 16 19:11:26.810415 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.810074    2566 scope.go:117] "RemoveContainer" containerID="a0ae73ccc8b5596963b8561d63bda12bdd9a953937ccbd1d5c0d7f88ca5fc57c"
Sep 16 19:11:26.828837 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.828811    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"]
Sep 16 19:11:26.831071 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:26.831048    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-40b85f95/scheduler-v06-threshold-migration-test-kserve-prefill-6444w4vjl"]
Sep 16 19:11:28.402559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:11:28.402514    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cc49762-8757-4926-8999-43004f0641c9" path="/var/lib/kubelet/pods/1cc49762-8757-4926-8999-43004f0641c9/volumes"
Sep 16 19:12:13.133044 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:13.132957    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:13.233599 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:13.233562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:20.125676 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:20.125631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:22.885456 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:22.885405    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:26.499022 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:26.498982    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:26.759707 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:26.759613    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:27.043089 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.042996    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"]
Sep 16 19:12:27.043363 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.043349    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cc49762-8757-4926-8999-43004f0641c9" containerName="main"
Sep 16 19:12:27.043412 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.043364    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cc49762-8757-4926-8999-43004f0641c9" containerName="main"
Sep 16 19:12:27.043454 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.043424    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cc49762-8757-4926-8999-43004f0641c9" containerName="main"
Sep 16 19:12:27.045811 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.045793    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.048738 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.048714    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-kserve-self-signed-certs\""
Sep 16 19:12:27.049583 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.049563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"kube-root-ca.crt\""
Sep 16 19:12:27.049741 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.049607    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"openshift-service-ca.crt\""
Sep 16 19:12:27.049741 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.049668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:27.049996 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.049972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"tokenizer-clean-path-test-epp-sa-dockercfg-hnp5x\""
Sep 16 19:12:27.050113 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.050066    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-cf47448b\"/\"default-dockercfg-tzf24\""
Sep 16 19:12:27.062494 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.062469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"]
Sep 16 19:12:27.137858 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.137811    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.138085 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.137939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg2s8\" (UniqueName: \"kubernetes.io/projected/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-kube-api-access-dg2s8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.138733 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.138704    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:27.238576 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.238520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.238576 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.238572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg2s8\" (UniqueName: \"kubernetes.io/projected/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-kube-api-access-dg2s8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.241056 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.241027    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-tls-certs\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.247974 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.247948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg2s8\" (UniqueName: \"kubernetes.io/projected/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-kube-api-access-dg2s8\") pod \"tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") " pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.357045 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.356947    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:27.644221 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:27.644197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"]
Sep 16 19:12:27.647177 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:12:27.647147    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36af6214_3bef_47c5_acf9_7d1d5bf0e7bc.slice/crio-85e8430f842493468adc9178f59d50157325884a48480620980721c4b01d5adf WatchSource:0}: Error finding container 85e8430f842493468adc9178f59d50157325884a48480620980721c4b01d5adf: Status 404 returned error can't find the container with id 85e8430f842493468adc9178f59d50157325884a48480620980721c4b01d5adf
Sep 16 19:12:28.119141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:28.119039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" event={"ID":"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc","Type":"ContainerStarted","Data":"ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33"}
Sep 16 19:12:28.119141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:28.119102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" event={"ID":"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc","Type":"ContainerStarted","Data":"85e8430f842493468adc9178f59d50157325884a48480620980721c4b01d5adf"}
Sep 16 19:12:28.119141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:28.119131    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:12:28.162969 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:28.162903    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podStartSLOduration=1.162867763 podStartE2EDuration="1.162867763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:28.159811343 +0000 UTC m=+1732.271607987" watchObservedRunningTime="2026-09-16 19:12:28.162867763 +0000 UTC m=+1732.274664405"
Sep 16 19:12:33.830684 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:33.830643    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:38.158677 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:38.158635    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:49.336801 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:49.336769    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-canary-service-backend-7b63470c_canary-v1-kserve-65d58ccc9-2b5js_4db5cb59-0d2d-48a0-92c2-e0e53e854108/main/0.log"
Sep 16 19:12:52.493640 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:52.493598    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:52.624343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:52.624309    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:12:59.126229 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:12:59.126196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:13:03.420436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.420392    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"]
Sep 16 19:13:03.426572 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.426535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.430310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.430284    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"openshift-service-ca.crt\""
Sep 16 19:13:03.430433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.430415    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"kube-root-ca.crt\""
Sep 16 19:13:03.431514 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.431497    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"default-dockercfg-7vtq8\""
Sep 16 19:13:03.431598 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.431534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-75f1ef44\"/\"test-div-v1-kserve-self-signed-certs\""
Sep 16 19:13:03.439428 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.439399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"]
Sep 16 19:13:03.524647 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.524610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:03.563738 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.563689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.563932 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.563777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tls-certs\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.563932 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.563818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-home\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.564043 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.563942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-dshm\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.564043 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.563986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-model-cache\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.564161 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.564104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2n9\" (UniqueName: \"kubernetes.io/projected/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-kube-api-access-mk2n9\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.664835 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.664779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-dshm\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.664835 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.664831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-model-cache\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665120 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.664849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2n9\" (UniqueName: \"kubernetes.io/projected/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-kube-api-access-mk2n9\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665120 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.664900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665120 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.664937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tls-certs\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665120 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.664972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-home\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665336 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.665281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-model-cache\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665415 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.665385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tmp-dir\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.665471 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.665416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-home\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.667235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.667214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-dshm\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.667528 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.667510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tls-certs\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.674822 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.674742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2n9\" (UniqueName: \"kubernetes.io/projected/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-kube-api-access-mk2n9\") pod \"test-div-v1-kserve-fb566d667-n9g6k\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") " pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:03.737526 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:03.737487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:04.016923 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:04.016874    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"]
Sep 16 19:13:04.017541 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:13:04.017518    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052b2b84_75ed_4ba6_8e2f_3a285ea2c975.slice/crio-0013329cdf8c2e9adbd948d1ff47f116141d9a6005eff4db7285e03b647d241d WatchSource:0}: Error finding container 0013329cdf8c2e9adbd948d1ff47f116141d9a6005eff4db7285e03b647d241d: Status 404 returned error can't find the container with id 0013329cdf8c2e9adbd948d1ff47f116141d9a6005eff4db7285e03b647d241d
Sep 16 19:13:04.314478 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:04.314380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k" event={"ID":"052b2b84-75ed-4ba6-8e2f-3a285ea2c975","Type":"ContainerStarted","Data":"a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2"}
Sep 16 19:13:04.314478 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:04.314441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k" event={"ID":"052b2b84-75ed-4ba6-8e2f-3a285ea2c975","Type":"ContainerStarted","Data":"0013329cdf8c2e9adbd948d1ff47f116141d9a6005eff4db7285e03b647d241d"}
Sep 16 19:13:04.341628 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:04.341561    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k" podStartSLOduration=1.341536753 podStartE2EDuration="1.341536753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:04.340106634 +0000 UTC m=+1768.451903276" watchObservedRunningTime="2026-09-16 19:13:04.341536753 +0000 UTC m=+1768.453333396"
Sep 16 19:13:04.738620 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:04.738578    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:13.737734 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:13.737688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:13.756318 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:13.756280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:14.378529 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:14.378498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:20.332916 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:20.332845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:25.410748 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:25.410705    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:25.510756 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:25.510713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:34.131175 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:34.131079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:34.310843 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:34.310806    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:34.487594 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:34.487555    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:34.875555 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:34.875450    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"]
Sep 16 19:13:34.952040 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:34.951993    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:35.730852 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:35.730817    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:39.237745 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:39.237696    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:13:49.894621 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:49.894579    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k" podUID="052b2b84-75ed-4ba6-8e2f-3a285ea2c975" containerName="main" containerID="cri-o://a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2" gracePeriod=45
Sep 16 19:13:50.143410 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.143380    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:50.295511 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295418    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tmp-dir\") pod \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") "
Sep 16 19:13:50.295511 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295453    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-dshm\") pod \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") "
Sep 16 19:13:50.295511 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295472    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tls-certs\") pod \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") "
Sep 16 19:13:50.295511 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-home\") pod \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") "
Sep 16 19:13:50.295511 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-model-cache\") pod \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") "
Sep 16 19:13:50.295917 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295682    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mk2n9\" (UniqueName: \"kubernetes.io/projected/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-kube-api-access-mk2n9\") pod \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\" (UID: \"052b2b84-75ed-4ba6-8e2f-3a285ea2c975\") "
Sep 16 19:13:50.295917 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295703    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "052b2b84-75ed-4ba6-8e2f-3a285ea2c975" (UID: "052b2b84-75ed-4ba6-8e2f-3a285ea2c975"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:50.295917 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295859    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-model-cache" (OuterVolumeSpecName: "model-cache") pod "052b2b84-75ed-4ba6-8e2f-3a285ea2c975" (UID: "052b2b84-75ed-4ba6-8e2f-3a285ea2c975"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:50.296090 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.295869    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-home" (OuterVolumeSpecName: "home") pod "052b2b84-75ed-4ba6-8e2f-3a285ea2c975" (UID: "052b2b84-75ed-4ba6-8e2f-3a285ea2c975"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:50.296090 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.296074    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.296171 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.296096    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.296171 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.296109    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.297864 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.297839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-dshm" (OuterVolumeSpecName: "dshm") pod "052b2b84-75ed-4ba6-8e2f-3a285ea2c975" (UID: "052b2b84-75ed-4ba6-8e2f-3a285ea2c975"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:50.297989 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.297926    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "052b2b84-75ed-4ba6-8e2f-3a285ea2c975" (UID: "052b2b84-75ed-4ba6-8e2f-3a285ea2c975"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:50.297989 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.297932    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-kube-api-access-mk2n9" (OuterVolumeSpecName: "kube-api-access-mk2n9") pod "052b2b84-75ed-4ba6-8e2f-3a285ea2c975" (UID: "052b2b84-75ed-4ba6-8e2f-3a285ea2c975"). InnerVolumeSpecName "kube-api-access-mk2n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:50.396627 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.396587    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk2n9\" (UniqueName: \"kubernetes.io/projected/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-kube-api-access-mk2n9\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.396841 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.396707    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.396841 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.396729    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/052b2b84-75ed-4ba6-8e2f-3a285ea2c975-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:13:50.556572 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.556470    2566 generic.go:356] "Generic (PLEG): container finished" podID="052b2b84-75ed-4ba6-8e2f-3a285ea2c975" containerID="a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2" exitCode=0
Sep 16 19:13:50.556572 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.556533    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"
Sep 16 19:13:50.556825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.556561    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k" event={"ID":"052b2b84-75ed-4ba6-8e2f-3a285ea2c975","Type":"ContainerDied","Data":"a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2"}
Sep 16 19:13:50.556825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.556628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k" event={"ID":"052b2b84-75ed-4ba6-8e2f-3a285ea2c975","Type":"ContainerDied","Data":"0013329cdf8c2e9adbd948d1ff47f116141d9a6005eff4db7285e03b647d241d"}
Sep 16 19:13:50.556825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.556659    2566 scope.go:117] "RemoveContainer" containerID="a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2"
Sep 16 19:13:50.566007 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.565984    2566 scope.go:117] "RemoveContainer" containerID="a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2"
Sep 16 19:13:50.566351 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:13:50.566321    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2\": container with ID starting with a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2 not found: ID does not exist" containerID="a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2"
Sep 16 19:13:50.566429 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.566366    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2"} err="failed to get container status \"a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2\": rpc error: code = NotFound desc = could not find container \"a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2\": container with ID starting with a3a1e79efb11df230f00f504c3ad04db147ddc721122b1466f06a5dbc3b915d2 not found: ID does not exist"
Sep 16 19:13:50.576095 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.576067    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"]
Sep 16 19:13:50.578419 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:50.578401    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-75f1ef44/test-div-v1-kserve-fb566d667-n9g6k"]
Sep 16 19:13:52.401845 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:13:52.401807    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052b2b84-75ed-4ba6-8e2f-3a285ea2c975" path="/var/lib/kubelet/pods/052b2b84-75ed-4ba6-8e2f-3a285ea2c975/volumes"
Sep 16 19:14:16.742559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:16.742515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:18.753331 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:18.753270    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:23.134494 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:23.134422    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:23.209438 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:23.209401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:35.106787 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.106749    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"]
Sep 16 19:14:35.107190 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.107144    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052b2b84-75ed-4ba6-8e2f-3a285ea2c975" containerName="main"
Sep 16 19:14:35.107190 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.107157    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="052b2b84-75ed-4ba6-8e2f-3a285ea2c975" containerName="main"
Sep 16 19:14:35.107263 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.107216    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="052b2b84-75ed-4ba6-8e2f-3a285ea2c975" containerName="main"
Sep 16 19:14:35.110472 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.110454    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.114582 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.114557    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"openshift-service-ca.crt\""
Sep 16 19:14:35.114582 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.114573    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"kube-root-ca.crt\""
Sep 16 19:14:35.114797 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.114669    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"test-div-v2-kserve-self-signed-certs\""
Sep 16 19:14:35.114797 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.114575    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-model-name-divergence-resolves-b3ec91d2\"/\"default-dockercfg-5xpjk\""
Sep 16 19:14:35.124540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.124468    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"]
Sep 16 19:14:35.160846 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.160805    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:35.189060 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.189018    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btpdl\" (UniqueName: \"kubernetes.io/projected/9349a1be-13bb-4771-a1ea-67e7dc6c8533-kube-api-access-btpdl\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.189060 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.189060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.189272 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.189138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.189272 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.189175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-home\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.189272 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.189204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.189383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.189277    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290078 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290039    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-home\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290451 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btpdl\" (UniqueName: \"kubernetes.io/projected/9349a1be-13bb-4771-a1ea-67e7dc6c8533-kube-api-access-btpdl\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290451 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290451 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290430    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-home\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290620 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290585    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-model-cache\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.290620 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.290586    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tmp-dir\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.292467 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.292440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-dshm\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.292650 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.292630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tls-certs\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.299134 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.299105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btpdl\" (UniqueName: \"kubernetes.io/projected/9349a1be-13bb-4771-a1ea-67e7dc6c8533-kube-api-access-btpdl\") pod \"test-div-v2-kserve-5f6fdb6b77-xsnhq\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") " pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.429635 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.429601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:35.704673 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.704645    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"]
Sep 16 19:14:35.706096 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:14:35.706069    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9349a1be_13bb_4771_a1ea_67e7dc6c8533.slice/crio-4f7c5c15b857da55909f37c658a74c0ee194b82131ca06a0fb37310e14a8124a WatchSource:0}: Error finding container 4f7c5c15b857da55909f37c658a74c0ee194b82131ca06a0fb37310e14a8124a: Status 404 returned error can't find the container with id 4f7c5c15b857da55909f37c658a74c0ee194b82131ca06a0fb37310e14a8124a
Sep 16 19:14:35.794837 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.794793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq" event={"ID":"9349a1be-13bb-4771-a1ea-67e7dc6c8533","Type":"ContainerStarted","Data":"cedc2128ea41341f104b08e057857e1fc9be2b1532f7d4930cd38782c0ba039c"}
Sep 16 19:14:35.795048 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.794850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq" event={"ID":"9349a1be-13bb-4771-a1ea-67e7dc6c8533","Type":"ContainerStarted","Data":"4f7c5c15b857da55909f37c658a74c0ee194b82131ca06a0fb37310e14a8124a"}
Sep 16 19:14:35.812760 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:35.812706    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq" podStartSLOduration=0.812689272 podStartE2EDuration="812.689272ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:35.812024276 +0000 UTC m=+1859.923820917" watchObservedRunningTime="2026-09-16 19:14:35.812689272 +0000 UTC m=+1859.924485913"
Sep 16 19:14:36.429839 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:36.429799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:45.430430 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:45.430390    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:45.443301 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:45.443265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:45.859106 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:45.859030    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:14:47.359102 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:47.359060    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:47.391314 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:47.391273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:47.421634 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:47.421597    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"]
Sep 16 19:14:48.978338 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:48.978292    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:14:49.126128 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:49.126085    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:14:59.125575 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:14:59.125484    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:02.443465 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:02.443398    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" containerID="cri-o://ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33" gracePeriod=45
Sep 16 19:15:09.126030 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:09.125988    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:09.126439 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:09.126093    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:15:11.855505 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:11.855468    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:11.997566 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:11.997524    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:12.097779 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:12.097741    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:12.623642 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:12.623609    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:12.766627 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:12.766593    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:12.980777 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:12.980737    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:13.138662 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:13.138619    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:13.884370 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:13.884328    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"]
Sep 16 19:15:13.924162 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:13.924126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:16.932342 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:16.932308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:18.930437 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:18.930403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:19.125510 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:19.125465    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:24.843376 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:24.843338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:24.914013 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:24.913979    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:28.940249 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:28.940182    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq" podUID="9349a1be-13bb-4771-a1ea-67e7dc6c8533" containerName="main" containerID="cri-o://cedc2128ea41341f104b08e057857e1fc9be2b1532f7d4930cd38782c0ba039c" gracePeriod=45
Sep 16 19:15:29.075522 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.075483    2566 generic.go:356] "Generic (PLEG): container finished" podID="9349a1be-13bb-4771-a1ea-67e7dc6c8533" containerID="cedc2128ea41341f104b08e057857e1fc9be2b1532f7d4930cd38782c0ba039c" exitCode=0
Sep 16 19:15:29.075675 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.075558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq" event={"ID":"9349a1be-13bb-4771-a1ea-67e7dc6c8533","Type":"ContainerDied","Data":"cedc2128ea41341f104b08e057857e1fc9be2b1532f7d4930cd38782c0ba039c"}
Sep 16 19:15:29.125654 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.125618    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:15:29.197954 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.197866    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:15:29.355266 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355230    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tls-certs\") pod \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") "
Sep 16 19:15:29.355489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-model-cache\") pod \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") "
Sep 16 19:15:29.355489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-dshm\") pod \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") "
Sep 16 19:15:29.355489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-home\") pod \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") "
Sep 16 19:15:29.355489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tmp-dir\") pod \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") "
Sep 16 19:15:29.355489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355382    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btpdl\" (UniqueName: \"kubernetes.io/projected/9349a1be-13bb-4771-a1ea-67e7dc6c8533-kube-api-access-btpdl\") pod \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\" (UID: \"9349a1be-13bb-4771-a1ea-67e7dc6c8533\") "
Sep 16 19:15:29.355792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355638    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-model-cache" (OuterVolumeSpecName: "model-cache") pod "9349a1be-13bb-4771-a1ea-67e7dc6c8533" (UID: "9349a1be-13bb-4771-a1ea-67e7dc6c8533"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:29.355792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355654    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-home" (OuterVolumeSpecName: "home") pod "9349a1be-13bb-4771-a1ea-67e7dc6c8533" (UID: "9349a1be-13bb-4771-a1ea-67e7dc6c8533"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:29.355792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.355671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9349a1be-13bb-4771-a1ea-67e7dc6c8533" (UID: "9349a1be-13bb-4771-a1ea-67e7dc6c8533"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:29.357804 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.357773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-dshm" (OuterVolumeSpecName: "dshm") pod "9349a1be-13bb-4771-a1ea-67e7dc6c8533" (UID: "9349a1be-13bb-4771-a1ea-67e7dc6c8533"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:29.357804 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.357788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9349a1be-13bb-4771-a1ea-67e7dc6c8533" (UID: "9349a1be-13bb-4771-a1ea-67e7dc6c8533"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:29.357982 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.357835    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9349a1be-13bb-4771-a1ea-67e7dc6c8533-kube-api-access-btpdl" (OuterVolumeSpecName: "kube-api-access-btpdl") pod "9349a1be-13bb-4771-a1ea-67e7dc6c8533" (UID: "9349a1be-13bb-4771-a1ea-67e7dc6c8533"). InnerVolumeSpecName "kube-api-access-btpdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:29.456087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.455996    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:29.456087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.456025    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:29.456087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.456036    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:29.456087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.456045    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:29.456087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.456054    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btpdl\" (UniqueName: \"kubernetes.io/projected/9349a1be-13bb-4771-a1ea-67e7dc6c8533-kube-api-access-btpdl\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:29.456087 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:29.456062    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9349a1be-13bb-4771-a1ea-67e7dc6c8533-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:30.083021 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:30.082990    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"
Sep 16 19:15:30.083448 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:30.082989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq" event={"ID":"9349a1be-13bb-4771-a1ea-67e7dc6c8533","Type":"ContainerDied","Data":"4f7c5c15b857da55909f37c658a74c0ee194b82131ca06a0fb37310e14a8124a"}
Sep 16 19:15:30.083448 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:30.083145    2566 scope.go:117] "RemoveContainer" containerID="cedc2128ea41341f104b08e057857e1fc9be2b1532f7d4930cd38782c0ba039c"
Sep 16 19:15:30.109014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:30.108983    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"]
Sep 16 19:15:30.115792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:30.115764    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-model-name-divergence-resolves-b3ec91d2/test-div-v2-kserve-5f6fdb6b77-xsnhq"]
Sep 16 19:15:30.402964 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:30.402928    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9349a1be-13bb-4771-a1ea-67e7dc6c8533" path="/var/lib/kubelet/pods/9349a1be-13bb-4771-a1ea-67e7dc6c8533/volumes"
Sep 16 19:15:31.147102 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:31.147070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:32.689051 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.689016    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:15:32.786067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.785979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-tls-certs\") pod \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") "
Sep 16 19:15:32.786067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.786042    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg2s8\" (UniqueName: \"kubernetes.io/projected/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-kube-api-access-dg2s8\") pod \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\" (UID: \"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc\") "
Sep 16 19:15:32.788283 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.788251    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-kube-api-access-dg2s8" (OuterVolumeSpecName: "kube-api-access-dg2s8") pod "36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" (UID: "36af6214-3bef-47c5-acf9-7d1d5bf0e7bc"). InnerVolumeSpecName "kube-api-access-dg2s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:32.788408 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.788284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" (UID: "36af6214-3bef-47c5-acf9-7d1d5bf0e7bc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:32.887544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.887502    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg2s8\" (UniqueName: \"kubernetes.io/projected/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-kube-api-access-dg2s8\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:32.887544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:32.887537    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:15:33.102126 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.102027    2566 generic.go:356] "Generic (PLEG): container finished" podID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerID="ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33" exitCode=0
Sep 16 19:15:33.102126 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.102079    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"
Sep 16 19:15:33.102126 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.102111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" event={"ID":"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc","Type":"ContainerDied","Data":"ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33"}
Sep 16 19:15:33.102427 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.102189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj" event={"ID":"36af6214-3bef-47c5-acf9-7d1d5bf0e7bc","Type":"ContainerDied","Data":"85e8430f842493468adc9178f59d50157325884a48480620980721c4b01d5adf"}
Sep 16 19:15:33.102427 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.102215    2566 scope.go:117] "RemoveContainer" containerID="ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33"
Sep 16 19:15:33.110828 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.110814    2566 scope.go:117] "RemoveContainer" containerID="ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33"
Sep 16 19:15:33.111135 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:15:33.111114    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33\": container with ID starting with ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33 not found: ID does not exist" containerID="ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33"
Sep 16 19:15:33.111196 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.111142    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33"} err="failed to get container status \"ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33\": rpc error: code = NotFound desc = could not find container \"ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33\": container with ID starting with ed7ac3967c7f0b1b97aa0588d82bd1deb0e06bb0a4b5144a445feac1372cec33 not found: ID does not exist"
Sep 16 19:15:33.124922 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.124894    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"]
Sep 16 19:15:33.130838 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:33.130814    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-cf47448b/tokenizer-clean-path-test-kserve-router-scheduler-58494b69v7hdj"]
Sep 16 19:15:34.382002 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:34.381964    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:34.402229 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:34.402195    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" path="/var/lib/kubelet/pods/36af6214-3bef-47c5-acf9-7d1d5bf0e7bc/volumes"
Sep 16 19:15:34.523564 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:34.523523    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:41.409719 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:41.409684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:47.333081 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333045    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"]
Sep 16 19:15:47.333485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333394    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main"
Sep 16 19:15:47.333485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333406    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main"
Sep 16 19:15:47.333485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333422    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9349a1be-13bb-4771-a1ea-67e7dc6c8533" containerName="main"
Sep 16 19:15:47.333485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333427    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9349a1be-13bb-4771-a1ea-67e7dc6c8533" containerName="main"
Sep 16 19:15:47.333626 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333488    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="36af6214-3bef-47c5-acf9-7d1d5bf0e7bc" containerName="main"
Sep 16 19:15:47.333626 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.333499    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9349a1be-13bb-4771-a1ea-67e7dc6c8533" containerName="main"
Sep 16 19:15:47.336504 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.336482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.340659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.340634    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"openshift-service-ca.crt\""
Sep 16 19:15:47.340930 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.340907    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"stop-v2-kserve-self-signed-certs\""
Sep 16 19:15:47.341864 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.341846    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"kube-root-ca.crt\""
Sep 16 19:15:47.341999 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.341862    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-force-stop-3e394e04\"/\"default-dockercfg-9b9gb\""
Sep 16 19:15:47.357355 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.357318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"]
Sep 16 19:15:47.402857 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.402823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.403034 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.402933    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-home\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.403082 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.403032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-dshm\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.403082 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.403049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-model-cache\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.403082 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.403065    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.403183 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.403094    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526zh\" (UniqueName: \"kubernetes.io/projected/349a88ad-5f65-4e7e-97bf-1cb6091587d9-kube-api-access-526zh\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.459069 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.459034    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:47.503787 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.503748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526zh\" (UniqueName: \"kubernetes.io/projected/349a88ad-5f65-4e7e-97bf-1cb6091587d9-kube-api-access-526zh\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.503787 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.503801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504023 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.503847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-home\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504023 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.503870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-dshm\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504023 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.503917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-model-cache\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504023 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.503962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504274 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.504257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tmp-dir\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504374 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.504354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-model-cache\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.504437 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.504375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-home\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.506137 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.506117    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-dshm\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.506429 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.506410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tls-certs\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.515810 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.515786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526zh\" (UniqueName: \"kubernetes.io/projected/349a88ad-5f65-4e7e-97bf-1cb6091587d9-kube-api-access-526zh\") pod \"stop-v2-kserve-7f68747c45-9x2lf\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") " pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.646937 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.646902    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:47.924679 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:47.924647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"]
Sep 16 19:15:47.926301 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:15:47.926269    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod349a88ad_5f65_4e7e_97bf_1cb6091587d9.slice/crio-75381f1461ca703142af73ec0c9793facaf44fdf50fda42d5ba97d54f8189b03 WatchSource:0}: Error finding container 75381f1461ca703142af73ec0c9793facaf44fdf50fda42d5ba97d54f8189b03: Status 404 returned error can't find the container with id 75381f1461ca703142af73ec0c9793facaf44fdf50fda42d5ba97d54f8189b03
Sep 16 19:15:48.182809 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:48.182719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" event={"ID":"349a88ad-5f65-4e7e-97bf-1cb6091587d9","Type":"ContainerStarted","Data":"d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e"}
Sep 16 19:15:48.182809 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:48.182778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" event={"ID":"349a88ad-5f65-4e7e-97bf-1cb6091587d9","Type":"ContainerStarted","Data":"75381f1461ca703142af73ec0c9793facaf44fdf50fda42d5ba97d54f8189b03"}
Sep 16 19:15:48.204338 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:48.204282    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" podStartSLOduration=1.204264536 podStartE2EDuration="1.204264536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:48.201854106 +0000 UTC m=+1932.313650760" watchObservedRunningTime="2026-09-16 19:15:48.204264536 +0000 UTC m=+1932.316061177"
Sep 16 19:15:48.555857 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:48.555767    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:48.647597 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:48.647554    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:48.775927 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:48.775868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:54.459379 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:54.459343    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:55.107262 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:55.107216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:15:57.647466 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:57.647423    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:57.660179 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:57.660140    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:15:58.249208 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:15:58.249171    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:16:00.234539 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:00.234498    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:00.308446 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:00.308402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:18.632434 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.632365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:18.955773 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.955740    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"]
Sep 16 19:16:18.959387 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.959364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:18.964926 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.964866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-epp-sa-dockercfg-5pdrj\""
Sep 16 19:16:18.965078 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.964962    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"default-dockercfg-85x4z\""
Sep 16 19:16:18.965078 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.964967    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"kube-root-ca.crt\""
Sep 16 19:16:18.965078 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.964892    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"tokenizer-migration-path-test-kserve-self-signed-certs\""
Sep 16 19:16:18.965376 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.965280    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-312caf2d\"/\"openshift-service-ca.crt\""
Sep 16 19:16:18.972305 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.972279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"]
Sep 16 19:16:18.978664 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:18.978638    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:19.074681 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.074639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.074919 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.074792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vr8d\" (UniqueName: \"kubernetes.io/projected/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-kube-api-access-9vr8d\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.078776 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.078748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:19.175656 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.175615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vr8d\" (UniqueName: \"kubernetes.io/projected/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-kube-api-access-9vr8d\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.175913 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.175667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.178155 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.178131    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-tls-certs\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.183724 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.183698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vr8d\" (UniqueName: \"kubernetes.io/projected/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-kube-api-access-9vr8d\") pod \"tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") " pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.271873 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.271781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:19.548824 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:19.548795    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"]
Sep 16 19:16:19.550932 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:16:19.550901    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ae768c_e7c5_4ab8_9eb8_3ea2be6cfa12.slice/crio-3e78524e1104a0d0c76775d5edaeebe8a8c2e876e4419a87ca4cb6aede191bc6 WatchSource:0}: Error finding container 3e78524e1104a0d0c76775d5edaeebe8a8c2e876e4419a87ca4cb6aede191bc6: Status 404 returned error can't find the container with id 3e78524e1104a0d0c76775d5edaeebe8a8c2e876e4419a87ca4cb6aede191bc6
Sep 16 19:16:20.354390 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:20.354351    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" event={"ID":"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12","Type":"ContainerStarted","Data":"c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94"}
Sep 16 19:16:20.354806 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:20.354410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" event={"ID":"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12","Type":"ContainerStarted","Data":"3e78524e1104a0d0c76775d5edaeebe8a8c2e876e4419a87ca4cb6aede191bc6"}
Sep 16 19:16:20.354806 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:20.354482    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:20.378197 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:20.378143    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podStartSLOduration=2.37812636 podStartE2EDuration="2.37812636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:20.375823758 +0000 UTC m=+1964.487620399" watchObservedRunningTime="2026-09-16 19:16:20.37812636 +0000 UTC m=+1964.489923000"
Sep 16 19:16:28.151155 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:28.151103    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:36.000449 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:36.000416    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"]
Sep 16 19:16:36.029592 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:36.029553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:51.019386 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.019321    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" podUID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" containerName="main" containerID="cri-o://d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e" gracePeriod=45
Sep 16 19:16:51.262255 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.262227    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:16:51.360903 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.360793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:16:51.366477 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-dshm\") pod \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") "
Sep 16 19:16:51.366579 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-home\") pod \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") "
Sep 16 19:16:51.366579 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tls-certs\") pod \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") "
Sep 16 19:16:51.366676 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366585    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tmp-dir\") pod \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") "
Sep 16 19:16:51.366676 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-model-cache\") pod \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") "
Sep 16 19:16:51.366810 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-526zh\" (UniqueName: \"kubernetes.io/projected/349a88ad-5f65-4e7e-97bf-1cb6091587d9-kube-api-access-526zh\") pod \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\" (UID: \"349a88ad-5f65-4e7e-97bf-1cb6091587d9\") "
Sep 16 19:16:51.366916 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366797    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-home" (OuterVolumeSpecName: "home") pod "349a88ad-5f65-4e7e-97bf-1cb6091587d9" (UID: "349a88ad-5f65-4e7e-97bf-1cb6091587d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:51.366977 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "349a88ad-5f65-4e7e-97bf-1cb6091587d9" (UID: "349a88ad-5f65-4e7e-97bf-1cb6091587d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:51.366977 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.366891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "349a88ad-5f65-4e7e-97bf-1cb6091587d9" (UID: "349a88ad-5f65-4e7e-97bf-1cb6091587d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:51.367141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.367125    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:51.367198 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.367147    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:51.367198 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.367161    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:51.369103 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.369077    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-dshm" (OuterVolumeSpecName: "dshm") pod "349a88ad-5f65-4e7e-97bf-1cb6091587d9" (UID: "349a88ad-5f65-4e7e-97bf-1cb6091587d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:51.369206 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.369124    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349a88ad-5f65-4e7e-97bf-1cb6091587d9-kube-api-access-526zh" (OuterVolumeSpecName: "kube-api-access-526zh") pod "349a88ad-5f65-4e7e-97bf-1cb6091587d9" (UID: "349a88ad-5f65-4e7e-97bf-1cb6091587d9"). InnerVolumeSpecName "kube-api-access-526zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:51.369206 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.369142    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "349a88ad-5f65-4e7e-97bf-1cb6091587d9" (UID: "349a88ad-5f65-4e7e-97bf-1cb6091587d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:51.467998 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.467962    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/349a88ad-5f65-4e7e-97bf-1cb6091587d9-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:51.467998 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.467998    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/349a88ad-5f65-4e7e-97bf-1cb6091587d9-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:51.468287 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.468013    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-526zh\" (UniqueName: \"kubernetes.io/projected/349a88ad-5f65-4e7e-97bf-1cb6091587d9-kube-api-access-526zh\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:16:51.530014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.529972    2566 generic.go:356] "Generic (PLEG): container finished" podID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" containerID="d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e" exitCode=0
Sep 16 19:16:51.530185 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.530020    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"
Sep 16 19:16:51.530185 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.530057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" event={"ID":"349a88ad-5f65-4e7e-97bf-1cb6091587d9","Type":"ContainerDied","Data":"d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e"}
Sep 16 19:16:51.530185 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.530117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" event={"ID":"349a88ad-5f65-4e7e-97bf-1cb6091587d9","Type":"ContainerDied","Data":"75381f1461ca703142af73ec0c9793facaf44fdf50fda42d5ba97d54f8189b03"}
Sep 16 19:16:51.530185 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.530140    2566 scope.go:117] "RemoveContainer" containerID="d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e"
Sep 16 19:16:51.539725 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.539709    2566 scope.go:117] "RemoveContainer" containerID="d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e"
Sep 16 19:16:51.540003 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:16:51.539984    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e\": container with ID starting with d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e not found: ID does not exist" containerID="d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e"
Sep 16 19:16:51.540071 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.540011    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e"} err="failed to get container status \"d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e\": rpc error: code = NotFound desc = could not find container \"d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e\": container with ID starting with d3348bdacee3c56d63745c5b13c89549d91f84e06fc56ba42339136ea63df04e not found: ID does not exist"
Sep 16 19:16:51.555667 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.555638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"]
Sep 16 19:16:51.566380 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:51.566359    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf"]
Sep 16 19:16:52.234606 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:52.234568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-force-stop-3e394e04/stop-v2-kserve-7f68747c45-9x2lf" podUID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:16:52.403301 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:52.403269    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" path="/var/lib/kubelet/pods/349a88ad-5f65-4e7e-97bf-1cb6091587d9/volumes"
Sep 16 19:16:56.308605 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:56.308562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:56.460505 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:56.460462    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:16:56.559032 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:16:56.558945    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:01.378431 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:01.378392    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:05.928596 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:05.928557    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:18.888580 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:18.888543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:38.817653 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:38.817618    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:43.713070 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.713033    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"]
Sep 16 19:17:43.713564 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.713420    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" containerName="main"
Sep 16 19:17:43.713564 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.713433    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" containerName="main"
Sep 16 19:17:43.713564 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.713494    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="349a88ad-5f65-4e7e-97bf-1cb6091587d9" containerName="main"
Sep 16 19:17:43.719042 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.719018    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.721955 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.721923    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"openshift-service-ca.crt\""
Sep 16 19:17:43.721955 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.721950    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"decom-v2-kserve-self-signed-certs\""
Sep 16 19:17:43.723045 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.723021    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"kube-root-ca.crt\""
Sep 16 19:17:43.723202 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.723148    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-decommission-203cd6d1\"/\"default-dockercfg-fldhq\""
Sep 16 19:17:43.731037 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.731006    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"]
Sep 16 19:17:43.760079 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.760046    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:43.850559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.850520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-tmp-dir\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.850559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.850562    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd253b4-fdfd-4bd5-8049-f049589e5817-tls-certs\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.850791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.850608    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-dshm\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.850791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.850650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-model-cache\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.850791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.850682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhjfv\" (UniqueName: \"kubernetes.io/projected/3fd253b4-fdfd-4bd5-8049-f049589e5817-kube-api-access-hhjfv\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.850791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.850747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-home\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.951654 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.951613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhjfv\" (UniqueName: \"kubernetes.io/projected/3fd253b4-fdfd-4bd5-8049-f049589e5817-kube-api-access-hhjfv\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.951865 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.951698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-home\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.951996 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.951856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-tmp-dir\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.951996 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.951925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd253b4-fdfd-4bd5-8049-f049589e5817-tls-certs\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.951996 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.951981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-dshm\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.952173 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.952032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-model-cache\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.952173 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.952128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-home\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.952269 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.952234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-tmp-dir\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.952424 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.952395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-model-cache\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.954339 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.954317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-dshm\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.954522 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.954505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd253b4-fdfd-4bd5-8049-f049589e5817-tls-certs\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:43.960411 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:43.960388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhjfv\" (UniqueName: \"kubernetes.io/projected/3fd253b4-fdfd-4bd5-8049-f049589e5817-kube-api-access-hhjfv\") pod \"decom-v2-kserve-77f867d787-dcmb4\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") " pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:44.034384 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:44.034287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:44.309259 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:44.309229    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"]
Sep 16 19:17:44.310951 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:17:44.310913    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd253b4_fdfd_4bd5_8049_f049589e5817.slice/crio-8ee9762099227abe6baafd4bc42db3fa3112ea8e57367e7dac69c7fd8bbe73f0 WatchSource:0}: Error finding container 8ee9762099227abe6baafd4bc42db3fa3112ea8e57367e7dac69c7fd8bbe73f0: Status 404 returned error can't find the container with id 8ee9762099227abe6baafd4bc42db3fa3112ea8e57367e7dac69c7fd8bbe73f0
Sep 16 19:17:44.563094 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:44.562999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:44.833503 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:44.833402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4" event={"ID":"3fd253b4-fdfd-4bd5-8049-f049589e5817","Type":"ContainerStarted","Data":"e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e"}
Sep 16 19:17:44.833503 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:44.833459    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4" event={"ID":"3fd253b4-fdfd-4bd5-8049-f049589e5817","Type":"ContainerStarted","Data":"8ee9762099227abe6baafd4bc42db3fa3112ea8e57367e7dac69c7fd8bbe73f0"}
Sep 16 19:17:44.854703 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:44.854638    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4" podStartSLOduration=1.854617954 podStartE2EDuration="1.854617954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:17:44.852157103 +0000 UTC m=+2048.963953779" watchObservedRunningTime="2026-09-16 19:17:44.854617954 +0000 UTC m=+2048.966414595"
Sep 16 19:17:45.035224 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:45.035183    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:45.347640 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:45.347600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:45.445972 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:45.445933    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:47.173469 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:47.173430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:47.218894 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:47.218844    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"]
Sep 16 19:17:47.278598 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:47.278559    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:17:51.361801 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:51.361760    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:17:54.035446 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:54.035399    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:54.048222 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:54.048178    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:17:54.899340 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:17:54.899306    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:18:01.361668 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:01.361568    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:02.239058 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:02.239016    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main" containerID="cri-o://c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94" gracePeriod=45
Sep 16 19:18:11.361208 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:11.361150    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:11.361622 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:11.361300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:18:14.212721 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:14.212681    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"]
Sep 16 19:18:14.247016 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:14.246977    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:21.360925 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:21.360860    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:29.232718 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.232653    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4" podUID="3fd253b4-fdfd-4bd5-8049-f049589e5817" containerName="main" containerID="cri-o://e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e" gracePeriod=45
Sep 16 19:18:29.504181 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.504146    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:18:29.568820 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.568776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-tmp-dir\") pod \"3fd253b4-fdfd-4bd5-8049-f049589e5817\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") "
Sep 16 19:18:29.569052 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.568848    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd253b4-fdfd-4bd5-8049-f049589e5817-tls-certs\") pod \"3fd253b4-fdfd-4bd5-8049-f049589e5817\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") "
Sep 16 19:18:29.569052 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.568869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-model-cache\") pod \"3fd253b4-fdfd-4bd5-8049-f049589e5817\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") "
Sep 16 19:18:29.569052 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.568947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhjfv\" (UniqueName: \"kubernetes.io/projected/3fd253b4-fdfd-4bd5-8049-f049589e5817-kube-api-access-hhjfv\") pod \"3fd253b4-fdfd-4bd5-8049-f049589e5817\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") "
Sep 16 19:18:29.569052 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-dshm\") pod \"3fd253b4-fdfd-4bd5-8049-f049589e5817\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") "
Sep 16 19:18:29.569289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569067    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-home\") pod \"3fd253b4-fdfd-4bd5-8049-f049589e5817\" (UID: \"3fd253b4-fdfd-4bd5-8049-f049589e5817\") "
Sep 16 19:18:29.569289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569166    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fd253b4-fdfd-4bd5-8049-f049589e5817" (UID: "3fd253b4-fdfd-4bd5-8049-f049589e5817"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:29.569289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fd253b4-fdfd-4bd5-8049-f049589e5817" (UID: "3fd253b4-fdfd-4bd5-8049-f049589e5817"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:29.569408 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569364    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-home" (OuterVolumeSpecName: "home") pod "3fd253b4-fdfd-4bd5-8049-f049589e5817" (UID: "3fd253b4-fdfd-4bd5-8049-f049589e5817"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:29.569408 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569385    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:29.569408 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.569406    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:29.571248 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.571226    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd253b4-fdfd-4bd5-8049-f049589e5817-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fd253b4-fdfd-4bd5-8049-f049589e5817" (UID: "3fd253b4-fdfd-4bd5-8049-f049589e5817"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:29.571324 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.571262    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-dshm" (OuterVolumeSpecName: "dshm") pod "3fd253b4-fdfd-4bd5-8049-f049589e5817" (UID: "3fd253b4-fdfd-4bd5-8049-f049589e5817"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:18:29.571365 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.571320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd253b4-fdfd-4bd5-8049-f049589e5817-kube-api-access-hhjfv" (OuterVolumeSpecName: "kube-api-access-hhjfv") pod "3fd253b4-fdfd-4bd5-8049-f049589e5817" (UID: "3fd253b4-fdfd-4bd5-8049-f049589e5817"). InnerVolumeSpecName "kube-api-access-hhjfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:29.670786 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.670741    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:29.670786 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.670778    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fd253b4-fdfd-4bd5-8049-f049589e5817-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:29.670786 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.670787    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fd253b4-fdfd-4bd5-8049-f049589e5817-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:29.670786 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:29.670798    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhjfv\" (UniqueName: \"kubernetes.io/projected/3fd253b4-fdfd-4bd5-8049-f049589e5817-kube-api-access-hhjfv\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:30.039629 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.039592    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:30.077669 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.077633    2566 generic.go:356] "Generic (PLEG): container finished" podID="3fd253b4-fdfd-4bd5-8049-f049589e5817" containerID="e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e" exitCode=0
Sep 16 19:18:30.077862 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.077693    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"
Sep 16 19:18:30.077862 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.077714    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4" event={"ID":"3fd253b4-fdfd-4bd5-8049-f049589e5817","Type":"ContainerDied","Data":"e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e"}
Sep 16 19:18:30.077862 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.077771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4" event={"ID":"3fd253b4-fdfd-4bd5-8049-f049589e5817","Type":"ContainerDied","Data":"8ee9762099227abe6baafd4bc42db3fa3112ea8e57367e7dac69c7fd8bbe73f0"}
Sep 16 19:18:30.077862 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.077803    2566 scope.go:117] "RemoveContainer" containerID="e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e"
Sep 16 19:18:30.091183 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.091159    2566 scope.go:117] "RemoveContainer" containerID="e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e"
Sep 16 19:18:30.091687 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:18:30.091647    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e\": container with ID starting with e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e not found: ID does not exist" containerID="e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e"
Sep 16 19:18:30.091805 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.091701    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e"} err="failed to get container status \"e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e\": rpc error: code = NotFound desc = could not find container \"e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e\": container with ID starting with e02c4ab6c6a8742701053a3403ebdc322c7a1f7e9a33a2ad028ed7ac82619a6e not found: ID does not exist"
Sep 16 19:18:30.116253 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.116213    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"]
Sep 16 19:18:30.142375 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.142323    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-decommission-203cd6d1/decom-v2-kserve-77f867d787-dcmb4"]
Sep 16 19:18:30.179947 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.179912    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:30.283457 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.283420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:30.403026 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:30.402981    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fd253b4-fdfd-4bd5-8049-f049589e5817" path="/var/lib/kubelet/pods/3fd253b4-fdfd-4bd5-8049-f049589e5817/volumes"
Sep 16 19:18:31.361504 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:31.361462    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:18:32.501002 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.500928    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:18:32.597811 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.597770    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-tls-certs\") pod \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") "
Sep 16 19:18:32.598037 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.597837    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vr8d\" (UniqueName: \"kubernetes.io/projected/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-kube-api-access-9vr8d\") pod \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\" (UID: \"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12\") "
Sep 16 19:18:32.600110 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.600072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" (UID: "b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:18:32.600110 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.600096    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-kube-api-access-9vr8d" (OuterVolumeSpecName: "kube-api-access-9vr8d") pod "b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" (UID: "b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12"). InnerVolumeSpecName "kube-api-access-9vr8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:18:32.699115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.699076    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:32.699115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:32.699107    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vr8d\" (UniqueName: \"kubernetes.io/projected/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12-kube-api-access-9vr8d\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:18:33.097561 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.097518    2566 generic.go:356] "Generic (PLEG): container finished" podID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerID="c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94" exitCode=0
Sep 16 19:18:33.097753 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.097570    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"
Sep 16 19:18:33.097753 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.097607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" event={"ID":"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12","Type":"ContainerDied","Data":"c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94"}
Sep 16 19:18:33.097753 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.097672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn" event={"ID":"b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12","Type":"ContainerDied","Data":"3e78524e1104a0d0c76775d5edaeebe8a8c2e876e4419a87ca4cb6aede191bc6"}
Sep 16 19:18:33.097753 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.097698    2566 scope.go:117] "RemoveContainer" containerID="c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94"
Sep 16 19:18:33.107547 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.107517    2566 scope.go:117] "RemoveContainer" containerID="c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94"
Sep 16 19:18:33.107927 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:18:33.107893    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94\": container with ID starting with c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94 not found: ID does not exist" containerID="c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94"
Sep 16 19:18:33.108012 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.107935    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94"} err="failed to get container status \"c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94\": rpc error: code = NotFound desc = could not find container \"c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94\": container with ID starting with c02f7117fa537e3e956fe5c07bf07d5a72dba348e33228a9f873c7d15cc08e94 not found: ID does not exist"
Sep 16 19:18:33.124573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.124534    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"]
Sep 16 19:18:33.129635 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:33.129600    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-312caf2d/tokenizer-migration-path-test-kserve-router-scheduler-5785xb2mn"]
Sep 16 19:18:34.402894 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:34.402852    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" path="/var/lib/kubelet/pods/b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12/volumes"
Sep 16 19:18:35.101430 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:35.101385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:37.641856 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:37.641813    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:37.737721 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:37.737676    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:44.726895 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:44.726836    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:44.840917 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:44.840865    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:50.774758 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:50.774716    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:50.863666 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:50.863619    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:51.818820 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.818783    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"]
Sep 16 19:18:51.819336 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.819312    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fd253b4-fdfd-4bd5-8049-f049589e5817" containerName="main"
Sep 16 19:18:51.819336 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.819333    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fd253b4-fdfd-4bd5-8049-f049589e5817" containerName="main"
Sep 16 19:18:51.819465 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.819349    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main"
Sep 16 19:18:51.819465 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.819357    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main"
Sep 16 19:18:51.819567 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.819490    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fd253b4-fdfd-4bd5-8049-f049589e5817" containerName="main"
Sep 16 19:18:51.819567 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.819504    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ae768c-e7c5-4ab8-9eb8-3ea2be6cfa12" containerName="main"
Sep 16 19:18:51.822110 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.822086    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:51.825176 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.825151    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"kube-root-ca.crt\""
Sep 16 19:18:51.825334 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.825243    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"default-dockercfg-9mt2q\""
Sep 16 19:18:51.826358 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.826337    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"leave-v2-kserve-self-signed-certs\""
Sep 16 19:18:51.826464 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.826355    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-leave-group-5b69d57a\"/\"openshift-service-ca.crt\""
Sep 16 19:18:51.838464 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.838432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"]
Sep 16 19:18:51.851433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.851400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:18:51.972072 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.972033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-dshm\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:51.972072 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.972074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-model-cache\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:51.972403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.972116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:51.972403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.972177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-home\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:51.972403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.972225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pxk\" (UniqueName: \"kubernetes.io/projected/598efdc4-5e94-4727-8e68-13bc1f54a5db-kube-api-access-26pxk\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:51.972403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:51.972270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598efdc4-5e94-4727-8e68-13bc1f54a5db-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.072902    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-dshm\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.072951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-model-cache\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.072977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.072998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-home\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073365 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.073023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pxk\" (UniqueName: \"kubernetes.io/projected/598efdc4-5e94-4727-8e68-13bc1f54a5db-kube-api-access-26pxk\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073365 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.073195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598efdc4-5e94-4727-8e68-13bc1f54a5db-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073496 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.073472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-model-cache\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073557 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.073493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-home\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.073607 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.073548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-tmp-dir\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.075476 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.075455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-dshm\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.075843 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.075814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598efdc4-5e94-4727-8e68-13bc1f54a5db-tls-certs\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.084639 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.084606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pxk\" (UniqueName: \"kubernetes.io/projected/598efdc4-5e94-4727-8e68-13bc1f54a5db-kube-api-access-26pxk\") pod \"leave-v2-kserve-556b4dddf9-rn4rr\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") " pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.132651 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.132615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:18:52.417792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:52.417761    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"]
Sep 16 19:18:52.419630 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:18:52.419600    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598efdc4_5e94_4727_8e68_13bc1f54a5db.slice/crio-70e6786d6777a58942b6578e413ec81cd04f6b8462965905ae9e67ab9134743f WatchSource:0}: Error finding container 70e6786d6777a58942b6578e413ec81cd04f6b8462965905ae9e67ab9134743f: Status 404 returned error can't find the container with id 70e6786d6777a58942b6578e413ec81cd04f6b8462965905ae9e67ab9134743f
Sep 16 19:18:53.207651 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:53.207605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr" event={"ID":"598efdc4-5e94-4727-8e68-13bc1f54a5db","Type":"ContainerStarted","Data":"592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483"}
Sep 16 19:18:53.208097 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:53.207673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr" event={"ID":"598efdc4-5e94-4727-8e68-13bc1f54a5db","Type":"ContainerStarted","Data":"70e6786d6777a58942b6578e413ec81cd04f6b8462965905ae9e67ab9134743f"}
Sep 16 19:18:53.230376 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:53.230322    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr" podStartSLOduration=2.230306209 podStartE2EDuration="2.230306209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:53.22953275 +0000 UTC m=+2117.341329403" watchObservedRunningTime="2026-09-16 19:18:53.230306209 +0000 UTC m=+2117.342102850"
Sep 16 19:18:54.133226 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:18:54.133179    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:19:02.133763 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:02.133712    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:19:02.146687 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:02.146653    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:19:02.265318 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:02.265265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:19:12.745138 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:12.745097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:16.317597 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:16.317555    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:19.169020 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:19.168981    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:19:19.278331 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:19:19.278297    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:05.889179 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:05.889136    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:06.231637 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.231548    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"]
Sep 16 19:20:06.235167 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.235144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.243097 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.243071    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"kube-root-ca.crt\""
Sep 16 19:20:06.244201 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.244179    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-kserve-self-signed-certs\""
Sep 16 19:20:06.244280 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.244186    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"openshift-service-ca.crt\""
Sep 16 19:20:06.245097 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.245078    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"tokenizer-template-args-test-epp-sa-dockercfg-9276p\""
Sep 16 19:20:06.245155 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.245098    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-0eb24714\"/\"default-dockercfg-c4ct4\""
Sep 16 19:20:06.258433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.258397    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"]
Sep 16 19:20:06.325012 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.324975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csbwd\" (UniqueName: \"kubernetes.io/projected/8c181651-d54e-4fce-a391-dec520119691-kube-api-access-csbwd\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.325207 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.325062    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c181651-d54e-4fce-a391-dec520119691-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.426259 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.426223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c181651-d54e-4fce-a391-dec520119691-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.426476 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.426315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csbwd\" (UniqueName: \"kubernetes.io/projected/8c181651-d54e-4fce-a391-dec520119691-kube-api-access-csbwd\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.429052 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.429020    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c181651-d54e-4fce-a391-dec520119691-tls-certs\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.441962 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.441930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csbwd\" (UniqueName: \"kubernetes.io/projected/8c181651-d54e-4fce-a391-dec520119691-kube-api-access-csbwd\") pod \"tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") " pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.546764 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.546659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:06.842190 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:06.842159    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"]
Sep 16 19:20:06.843247 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:20:06.843211    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c181651_d54e_4fce_a391_dec520119691.slice/crio-b328d847a1282db5cb71b2f98841e0494c0ebe94796ab8e025d4d500a16b9545 WatchSource:0}: Error finding container b328d847a1282db5cb71b2f98841e0494c0ebe94796ab8e025d4d500a16b9545: Status 404 returned error can't find the container with id b328d847a1282db5cb71b2f98841e0494c0ebe94796ab8e025d4d500a16b9545
Sep 16 19:20:07.152649 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:07.152613    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:07.612038 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:07.611939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" event={"ID":"8c181651-d54e-4fce-a391-dec520119691","Type":"ContainerStarted","Data":"a2e505c869736fbacd65e325cfb7f8759002935d2768ad9b6d46fc5781dc7795"}
Sep 16 19:20:07.612038 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:07.611992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" event={"ID":"8c181651-d54e-4fce-a391-dec520119691","Type":"ContainerStarted","Data":"b328d847a1282db5cb71b2f98841e0494c0ebe94796ab8e025d4d500a16b9545"}
Sep 16 19:20:07.612038 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:07.612015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:07.636980 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:07.636913    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podStartSLOduration=1.636874781 podStartE2EDuration="1.636874781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:20:07.633958386 +0000 UTC m=+2191.745755024" watchObservedRunningTime="2026-09-16 19:20:07.636874781 +0000 UTC m=+2191.748671422"
Sep 16 19:20:10.090050 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:10.090008    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:35.246411 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:35.246365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:38.620465 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:38.620433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:20:45.846572 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:45.846529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:45.871695 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:45.871656    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"]
Sep 16 19:20:45.946116 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:45.946073    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:48.620890 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:48.620844    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:20:48.734192 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:48.734151    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:48.839294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:48.839259    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:48.937056 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:48.937011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:49.311470 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:49.311373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:49.411540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:49.411504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:52.260388 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:52.260349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:52.355550 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:52.355513    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"]
Sep 16 19:20:52.399598 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:52.399563    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:53.807669 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:53.807634    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:20:58.620864 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:20:58.620769    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:00.891648 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:00.891586    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main" containerID="cri-o://a2e505c869736fbacd65e325cfb7f8759002935d2768ad9b6d46fc5781dc7795" gracePeriod=45
Sep 16 19:21:07.375741 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.375694    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr" podUID="598efdc4-5e94-4727-8e68-13bc1f54a5db" containerName="main" containerID="cri-o://592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483" gracePeriod=45
Sep 16 19:21:07.504333 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.504278    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:07.653568 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.653536    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:21:07.665798 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.665764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-home\") pod \"598efdc4-5e94-4727-8e68-13bc1f54a5db\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") "
Sep 16 19:21:07.666013 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.665821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26pxk\" (UniqueName: \"kubernetes.io/projected/598efdc4-5e94-4727-8e68-13bc1f54a5db-kube-api-access-26pxk\") pod \"598efdc4-5e94-4727-8e68-13bc1f54a5db\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") "
Sep 16 19:21:07.666013 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.665846    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-dshm\") pod \"598efdc4-5e94-4727-8e68-13bc1f54a5db\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") "
Sep 16 19:21:07.666013 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.665869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598efdc4-5e94-4727-8e68-13bc1f54a5db-tls-certs\") pod \"598efdc4-5e94-4727-8e68-13bc1f54a5db\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") "
Sep 16 19:21:07.666013 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.665963    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-model-cache\") pod \"598efdc4-5e94-4727-8e68-13bc1f54a5db\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") "
Sep 16 19:21:07.666264 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.666037    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-tmp-dir\") pod \"598efdc4-5e94-4727-8e68-13bc1f54a5db\" (UID: \"598efdc4-5e94-4727-8e68-13bc1f54a5db\") "
Sep 16 19:21:07.666264 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.666144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-home" (OuterVolumeSpecName: "home") pod "598efdc4-5e94-4727-8e68-13bc1f54a5db" (UID: "598efdc4-5e94-4727-8e68-13bc1f54a5db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.666264 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.666235    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-model-cache" (OuterVolumeSpecName: "model-cache") pod "598efdc4-5e94-4727-8e68-13bc1f54a5db" (UID: "598efdc4-5e94-4727-8e68-13bc1f54a5db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.666421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.666359    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.666421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.666387    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.666421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.666358    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "598efdc4-5e94-4727-8e68-13bc1f54a5db" (UID: "598efdc4-5e94-4727-8e68-13bc1f54a5db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.668297 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.668255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598efdc4-5e94-4727-8e68-13bc1f54a5db-kube-api-access-26pxk" (OuterVolumeSpecName: "kube-api-access-26pxk") pod "598efdc4-5e94-4727-8e68-13bc1f54a5db" (UID: "598efdc4-5e94-4727-8e68-13bc1f54a5db"). InnerVolumeSpecName "kube-api-access-26pxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:07.668424 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.668301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598efdc4-5e94-4727-8e68-13bc1f54a5db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "598efdc4-5e94-4727-8e68-13bc1f54a5db" (UID: "598efdc4-5e94-4727-8e68-13bc1f54a5db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:07.668424 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.668349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-dshm" (OuterVolumeSpecName: "dshm") pod "598efdc4-5e94-4727-8e68-13bc1f54a5db" (UID: "598efdc4-5e94-4727-8e68-13bc1f54a5db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:07.767145 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.767103    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.767145 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.767137    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/598efdc4-5e94-4727-8e68-13bc1f54a5db-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.767385 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.767168    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/598efdc4-5e94-4727-8e68-13bc1f54a5db-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.767385 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.767177    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pxk\" (UniqueName: \"kubernetes.io/projected/598efdc4-5e94-4727-8e68-13bc1f54a5db-kube-api-access-26pxk\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:07.959651 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.959616    2566 generic.go:356] "Generic (PLEG): container finished" podID="598efdc4-5e94-4727-8e68-13bc1f54a5db" containerID="592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483" exitCode=0
Sep 16 19:21:07.959840 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.959661    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"
Sep 16 19:21:07.959840 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.959687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr" event={"ID":"598efdc4-5e94-4727-8e68-13bc1f54a5db","Type":"ContainerDied","Data":"592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483"}
Sep 16 19:21:07.959840 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.959760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr" event={"ID":"598efdc4-5e94-4727-8e68-13bc1f54a5db","Type":"ContainerDied","Data":"70e6786d6777a58942b6578e413ec81cd04f6b8462965905ae9e67ab9134743f"}
Sep 16 19:21:07.959840 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.959781    2566 scope.go:117] "RemoveContainer" containerID="592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483"
Sep 16 19:21:07.969444 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.969422    2566 scope.go:117] "RemoveContainer" containerID="592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483"
Sep 16 19:21:07.969775 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:21:07.969754    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483\": container with ID starting with 592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483 not found: ID does not exist" containerID="592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483"
Sep 16 19:21:07.969821 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.969787    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483"} err="failed to get container status \"592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483\": rpc error: code = NotFound desc = could not find container \"592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483\": container with ID starting with 592a42e473534c4e65ec762bb0a4aaa0a968a09343234cf5e129437daa279483 not found: ID does not exist"
Sep 16 19:21:07.985605 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.985566    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"]
Sep 16 19:21:07.991011 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:07.990976    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-leave-group-5b69d57a/leave-v2-kserve-556b4dddf9-rn4rr"]
Sep 16 19:21:08.403420 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:08.403379    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="598efdc4-5e94-4727-8e68-13bc1f54a5db" path="/var/lib/kubelet/pods/598efdc4-5e94-4727-8e68-13bc1f54a5db/volumes"
Sep 16 19:21:08.620544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:08.620500    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:08.620719 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:08.620635    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:21:12.843948 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:12.843909    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:18.621274 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:18.621232    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:22.935821 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:22.935774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:28.621126 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:28.621081    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:21:31.090135 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.090089    2566 generic.go:356] "Generic (PLEG): container finished" podID="8c181651-d54e-4fce-a391-dec520119691" containerID="a2e505c869736fbacd65e325cfb7f8759002935d2768ad9b6d46fc5781dc7795" exitCode=0
Sep 16 19:21:31.090530 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.090158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" event={"ID":"8c181651-d54e-4fce-a391-dec520119691","Type":"ContainerDied","Data":"a2e505c869736fbacd65e325cfb7f8759002935d2768ad9b6d46fc5781dc7795"}
Sep 16 19:21:31.141940 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.141907    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:21:31.282170 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.282077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c181651-d54e-4fce-a391-dec520119691-tls-certs\") pod \"8c181651-d54e-4fce-a391-dec520119691\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") "
Sep 16 19:21:31.282170 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.282168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-csbwd\" (UniqueName: \"kubernetes.io/projected/8c181651-d54e-4fce-a391-dec520119691-kube-api-access-csbwd\") pod \"8c181651-d54e-4fce-a391-dec520119691\" (UID: \"8c181651-d54e-4fce-a391-dec520119691\") "
Sep 16 19:21:31.284414 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.284382    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c181651-d54e-4fce-a391-dec520119691-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8c181651-d54e-4fce-a391-dec520119691" (UID: "8c181651-d54e-4fce-a391-dec520119691"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:31.284551 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.284454    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c181651-d54e-4fce-a391-dec520119691-kube-api-access-csbwd" (OuterVolumeSpecName: "kube-api-access-csbwd") pod "8c181651-d54e-4fce-a391-dec520119691" (UID: "8c181651-d54e-4fce-a391-dec520119691"). InnerVolumeSpecName "kube-api-access-csbwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:31.383550 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.383502    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c181651-d54e-4fce-a391-dec520119691-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:31.383550 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.383540    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csbwd\" (UniqueName: \"kubernetes.io/projected/8c181651-d54e-4fce-a391-dec520119691-kube-api-access-csbwd\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:21:31.434611 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:31.434552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:32.096500 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:32.096461    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb" event={"ID":"8c181651-d54e-4fce-a391-dec520119691","Type":"ContainerDied","Data":"b328d847a1282db5cb71b2f98841e0494c0ebe94796ab8e025d4d500a16b9545"}
Sep 16 19:21:32.096993 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:32.096526    2566 scope.go:117] "RemoveContainer" containerID="a2e505c869736fbacd65e325cfb7f8759002935d2768ad9b6d46fc5781dc7795"
Sep 16 19:21:32.096993 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:32.096471    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"
Sep 16 19:21:32.128483 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:32.128444    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"]
Sep 16 19:21:32.136309 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:32.136276    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-0eb24714/tokenizer-template-args-test-kserve-router-scheduler-8cd7dzhpwb"]
Sep 16 19:21:32.402370 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:32.402315    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c181651-d54e-4fce-a391-dec520119691" path="/var/lib/kubelet/pods/8c181651-d54e-4fce-a391-dec520119691/volumes"
Sep 16 19:21:34.568326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:34.568288    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:34.681856 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:34.681819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:37.997571 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:37.997536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:38.094253 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:38.094224    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:41.205906 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:41.205840    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:47.757745 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:47.757706    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:21:47.859382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:21:47.859336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:27.813297 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:27.813256    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:28.066735 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.066645    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"]
Sep 16 19:22:28.067046 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.067031    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main"
Sep 16 19:22:28.067102 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.067048    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main"
Sep 16 19:22:28.067102 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.067057    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="598efdc4-5e94-4727-8e68-13bc1f54a5db" containerName="main"
Sep 16 19:22:28.067102 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.067062    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="598efdc4-5e94-4727-8e68-13bc1f54a5db" containerName="main"
Sep 16 19:22:28.067200 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.067132    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="598efdc4-5e94-4727-8e68-13bc1f54a5db" containerName="main"
Sep 16 19:22:28.067200 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.067140    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c181651-d54e-4fce-a391-dec520119691" containerName="main"
Sep 16 19:22:28.069904 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.069855    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.073634 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.073589    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisv5c7e67b6c51568d1d6d13829a9337f2a-kserve-self-signed-certs\""
Sep 16 19:22:28.074762 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.074732    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"openshift-service-ca.crt\""
Sep 16 19:22:28.074932 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.074744    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"kube-root-ca.crt\""
Sep 16 19:22:28.074932 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.074779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"llmisvc-router-managed-test-llm-e45d1f79-epp-sa-dockercfg-zlvzr\""
Sep 16 19:22:28.075212 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.075195    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-350756ca\"/\"default-dockercfg-c2nv5\""
Sep 16 19:22:28.089269 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.089232    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"]
Sep 16 19:22:28.118498 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.118456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:22:28.164015 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.163974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvd2f\" (UniqueName: \"kubernetes.io/projected/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-kube-api-access-zvd2f\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.164015 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.164021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.265298 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.265252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvd2f\" (UniqueName: \"kubernetes.io/projected/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-kube-api-access-zvd2f\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.265298 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.265298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.267981 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.267951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-tls-certs\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.274370 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.274340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvd2f\" (UniqueName: \"kubernetes.io/projected/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-kube-api-access-zvd2f\") pod \"llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") " pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.383312 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.383204    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:28.673541 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:28.673506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"]
Sep 16 19:22:28.674944 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:22:28.674911    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efcce4b_20f3_465f_b2db_92f1fc3bfc77.slice/crio-893dcd93b06cebcb5732637c1a50e0029b8765a0003792a28c9ba7c97c997ab0 WatchSource:0}: Error finding container 893dcd93b06cebcb5732637c1a50e0029b8765a0003792a28c9ba7c97c997ab0: Status 404 returned error can't find the container with id 893dcd93b06cebcb5732637c1a50e0029b8765a0003792a28c9ba7c97c997ab0
Sep 16 19:22:29.405959 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:29.405917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" event={"ID":"9efcce4b-20f3-465f-b2db-92f1fc3bfc77","Type":"ContainerStarted","Data":"491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508"}
Sep 16 19:22:29.406383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:29.405978    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" event={"ID":"9efcce4b-20f3-465f-b2db-92f1fc3bfc77","Type":"ContainerStarted","Data":"893dcd93b06cebcb5732637c1a50e0029b8765a0003792a28c9ba7c97c997ab0"}
Sep 16 19:22:29.406383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:29.406004    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:22:46.249385 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:22:46.249341    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:00.413495 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:00.413460    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:23:00.435483 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:00.435408    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podStartSLOduration=32.435391736 podStartE2EDuration="32.435391736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:29.448941987 +0000 UTC m=+2333.560738626" watchObservedRunningTime="2026-09-16 19:23:00.435391736 +0000 UTC m=+2364.547188377"
Sep 16 19:23:10.012297 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:10.012253    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"]
Sep 16 19:23:10.035838 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:10.035798    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:10.106581 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:10.106544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:10.413812 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:10.413766    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:20.413507 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:20.413461    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:25.032774 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:25.032722    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main" containerID="cri-o://491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508" gracePeriod=45
Sep 16 19:23:30.413463 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:30.413420    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:30.413944 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:30.413539    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:23:39.075677 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:39.075636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:39.287216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:39.287178    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:40.413477 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:40.413435    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:40.653489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:40.653447    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:41.603900 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:41.603834    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:42.010677 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:42.010639    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:44.213015 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:44.212969    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:50.413948 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:50.413902    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:23:53.653028 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:53.652990    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:55.296773 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.296697    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:23:55.376938 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.376867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-tls-certs\") pod \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") "
Sep 16 19:23:55.377146 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.377007    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvd2f\" (UniqueName: \"kubernetes.io/projected/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-kube-api-access-zvd2f\") pod \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\" (UID: \"9efcce4b-20f3-465f-b2db-92f1fc3bfc77\") "
Sep 16 19:23:55.379306 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.379267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9efcce4b-20f3-465f-b2db-92f1fc3bfc77" (UID: "9efcce4b-20f3-465f-b2db-92f1fc3bfc77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:55.379442 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.379302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-kube-api-access-zvd2f" (OuterVolumeSpecName: "kube-api-access-zvd2f") pod "9efcce4b-20f3-465f-b2db-92f1fc3bfc77" (UID: "9efcce4b-20f3-465f-b2db-92f1fc3bfc77"). InnerVolumeSpecName "kube-api-access-zvd2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:55.478014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.477970    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.478014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.478007    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvd2f\" (UniqueName: \"kubernetes.io/projected/9efcce4b-20f3-465f-b2db-92f1fc3bfc77-kube-api-access-zvd2f\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.880865 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.880829    2566 generic.go:356] "Generic (PLEG): container finished" podID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerID="491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508" exitCode=0
Sep 16 19:23:55.881067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.880903    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"
Sep 16 19:23:55.881067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.880965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" event={"ID":"9efcce4b-20f3-465f-b2db-92f1fc3bfc77","Type":"ContainerDied","Data":"491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508"}
Sep 16 19:23:55.881067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.881019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff" event={"ID":"9efcce4b-20f3-465f-b2db-92f1fc3bfc77","Type":"ContainerDied","Data":"893dcd93b06cebcb5732637c1a50e0029b8765a0003792a28c9ba7c97c997ab0"}
Sep 16 19:23:55.881067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.881046    2566 scope.go:117] "RemoveContainer" containerID="491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508"
Sep 16 19:23:55.890636 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.890614    2566 scope.go:117] "RemoveContainer" containerID="491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508"
Sep 16 19:23:55.890976 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:23:55.890955    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508\": container with ID starting with 491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508 not found: ID does not exist" containerID="491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508"
Sep 16 19:23:55.891035 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.890986    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508"} err="failed to get container status \"491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508\": rpc error: code = NotFound desc = could not find container \"491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508\": container with ID starting with 491c86877be3636c9deb7e5b3c7a856b25726ba80fef0deb6d89bbacb7e91508 not found: ID does not exist"
Sep 16 19:23:55.909259 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.909219    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"]
Sep 16 19:23:55.916367 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:55.916333    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-350756ca/llmisvc5c7e67b6c51568d1d6d13829a9337f2a-kserve-router-schejrzff"]
Sep 16 19:23:56.317890 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.317830    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"]
Sep 16 19:23:56.318327 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.318280    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main"
Sep 16 19:23:56.318327 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.318297    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main"
Sep 16 19:23:56.318457 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.318375    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" containerName="main"
Sep 16 19:23:56.322099 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.322070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.325537 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.325510    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"late-v1-kserve-self-signed-certs\""
Sep 16 19:23:56.325742 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.325719    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"openshift-service-ca.crt\""
Sep 16 19:23:56.325959 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.325940    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-late-join-483adbac\"/\"kube-root-ca.crt\""
Sep 16 19:23:56.326860 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.326844    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-late-join-483adbac\"/\"default-dockercfg-d6lsx\""
Sep 16 19:23:56.336597 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.336562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"]
Sep 16 19:23:56.344056 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.344024    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:23:56.388792 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.388756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-model-cache\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.389005 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.388808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-dshm\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.389005 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.388856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.389005 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.388915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-home\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.389005 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.388965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwqh\" (UniqueName: \"kubernetes.io/projected/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-kube-api-access-lpwqh\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.389150 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.389024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.403388 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.403347    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9efcce4b-20f3-465f-b2db-92f1fc3bfc77" path="/var/lib/kubelet/pods/9efcce4b-20f3-465f-b2db-92f1fc3bfc77/volumes"
Sep 16 19:23:56.489672 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.489632    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-dshm\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.489672 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.489675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.489962 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.489797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-home\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.489962 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.489895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwqh\" (UniqueName: \"kubernetes.io/projected/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-kube-api-access-lpwqh\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.489962 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.489939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.490135 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.489989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-model-cache\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.490273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.490249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-home\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.490340 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.490290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-model-cache\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.490411 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.490388    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tmp-dir\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.492006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.491987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-dshm\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.492432 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.492412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tls-certs\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.499764 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.499738    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwqh\" (UniqueName: \"kubernetes.io/projected/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-kube-api-access-lpwqh\") pod \"late-v1-kserve-56cbc8456c-zk2bn\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") " pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.633924 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.633788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:23:56.915579 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:56.915548    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"]
Sep 16 19:23:56.917323 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:23:56.917279    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7754155e_56bf_48f6_b1bb_f5d1f83fbea1.slice/crio-18166b9704492092fc8923029b020825ab614bdabcc0bff18ce1b9a3527d33ff WatchSource:0}: Error finding container 18166b9704492092fc8923029b020825ab614bdabcc0bff18ce1b9a3527d33ff: Status 404 returned error can't find the container with id 18166b9704492092fc8923029b020825ab614bdabcc0bff18ce1b9a3527d33ff
Sep 16 19:23:57.895610 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:57.895563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" event={"ID":"7754155e-56bf-48f6-b1bb-f5d1f83fbea1","Type":"ContainerStarted","Data":"c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52"}
Sep 16 19:23:57.896156 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:57.895632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" event={"ID":"7754155e-56bf-48f6-b1bb-f5d1f83fbea1","Type":"ContainerStarted","Data":"18166b9704492092fc8923029b020825ab614bdabcc0bff18ce1b9a3527d33ff"}
Sep 16 19:23:57.916150 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:57.916081    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" podStartSLOduration=1.916060839 podStartE2EDuration="1.916060839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:57.913198028 +0000 UTC m=+2422.024994660" watchObservedRunningTime="2026-09-16 19:23:57.916060839 +0000 UTC m=+2422.027857479"
Sep 16 19:23:58.634267 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:23:58.634170    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:24:00.941216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:00.941173    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:01.041549 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:01.041513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:06.634208 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:06.634165    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:24:06.647347 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:06.647312    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:24:06.956926 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:06.956864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:24:07.468745 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:07.468705    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:13.596609 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.596553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:13.637834 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.637788    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"]
Sep 16 19:24:13.641172 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.641149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.644298 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.644271    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"default-dockercfg-cjx6p\""
Sep 16 19:24:13.644802 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.644782    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"openshift-service-ca.crt\""
Sep 16 19:24:13.645438 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.645413    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisve55ae740357a3a31a27cdb8b66ffe20f-kserve-self-signed-certs\""
Sep 16 19:24:13.645545 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.645449    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"kube-root-ca.crt\""
Sep 16 19:24:13.662988 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.662949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"]
Sep 16 19:24:13.728428 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.728385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:13.744772 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.744733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.744772 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.744779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ljr\" (UniqueName: \"kubernetes.io/projected/072ed10c-a128-437b-9f9b-4986bce70739-kube-api-access-k6ljr\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.745038 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.744801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072ed10c-a128-437b-9f9b-4986bce70739-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.745038 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.744932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.745115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.745041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.745115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.745074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.845995 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.845951    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ljr\" (UniqueName: \"kubernetes.io/projected/072ed10c-a128-437b-9f9b-4986bce70739-kube-api-access-k6ljr\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.845995 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.845997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072ed10c-a128-437b-9f9b-4986bce70739-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846086    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846499 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846499 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-tmp-dir\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-home\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.846659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.846603    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-model-cache\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.848590 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.848569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-dshm\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.848830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.848808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072ed10c-a128-437b-9f9b-4986bce70739-tls-certs\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.857939 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.857908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ljr\" (UniqueName: \"kubernetes.io/projected/072ed10c-a128-437b-9f9b-4986bce70739-kube-api-access-k6ljr\") pod \"llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") " pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:13.953478 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:13.953434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:14.011633 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:14.011601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:14.234666 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:14.234628    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"]
Sep 16 19:24:14.236048 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:24:14.236016    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072ed10c_a128_437b_9f9b_4986bce70739.slice/crio-698ecd34a1b1f1ca37362bb3c57ffb30f4b9fb5f2eed3f7a504b6b0c14b92b13 WatchSource:0}: Error finding container 698ecd34a1b1f1ca37362bb3c57ffb30f4b9fb5f2eed3f7a504b6b0c14b92b13: Status 404 returned error can't find the container with id 698ecd34a1b1f1ca37362bb3c57ffb30f4b9fb5f2eed3f7a504b6b0c14b92b13
Sep 16 19:24:14.992061 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:14.992022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" event={"ID":"072ed10c-a128-437b-9f9b-4986bce70739","Type":"ContainerStarted","Data":"aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"}
Sep 16 19:24:14.992481 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:14.992082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" event={"ID":"072ed10c-a128-437b-9f9b-4986bce70739","Type":"ContainerStarted","Data":"698ecd34a1b1f1ca37362bb3c57ffb30f4b9fb5f2eed3f7a504b6b0c14b92b13"}
Sep 16 19:24:15.019512 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:15.019446    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" podStartSLOduration=2.01942838 podStartE2EDuration="2.01942838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:15.018254867 +0000 UTC m=+2439.130051509" watchObservedRunningTime="2026-09-16 19:24:15.01942838 +0000 UTC m=+2439.131225020"
Sep 16 19:24:15.953949 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:15.953904    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:18.732380 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:18.732344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:21.832197 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:21.832156    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:23.954318 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:23.954275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:23.967519 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:23.967483    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:24.052272 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:24.052236    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:24:27.693394 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:27.693355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:27.765298 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:27.765254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:30.464017 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:30.463978    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:39.446226 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:39.446174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:24:45.761522 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:24:45.761482    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:20.253450 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:20.253408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:23.567721 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.567682    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:23.772139 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.772098    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"]
Sep 16 19:25:23.775648 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.775618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:23.779787 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.779759    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Sep 16 19:25:23.779961 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.779759    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-84370a53\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-kc59j\""
Sep 16 19:25:23.787655 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.787628    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"]
Sep 16 19:25:23.800073 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.800026    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:23.878183 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.878089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cd12-6461-4916-900d-61fcfa222839-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:23.878343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.878179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmzrt\" (UniqueName: \"kubernetes.io/projected/4a67cd12-6461-4916-900d-61fcfa222839-kube-api-access-vmzrt\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:23.979403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.979367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmzrt\" (UniqueName: \"kubernetes.io/projected/4a67cd12-6461-4916-900d-61fcfa222839-kube-api-access-vmzrt\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:23.979593 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.979447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cd12-6461-4916-900d-61fcfa222839-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:23.982039 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.982008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cd12-6461-4916-900d-61fcfa222839-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:23.988051 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:23.988017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmzrt\" (UniqueName: \"kubernetes.io/projected/4a67cd12-6461-4916-900d-61fcfa222839-kube-api-access-vmzrt\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") " pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:24.086441 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:24.086392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:24.367267 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:24.367239    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"]
Sep 16 19:25:24.369076 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:25:24.369041    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a67cd12_6461_4916_900d_61fcfa222839.slice/crio-ce223dfbc93652ce2b90498e6c64ef01ecd747a6fe072334bc819daab69b83db WatchSource:0}: Error finding container ce223dfbc93652ce2b90498e6c64ef01ecd747a6fe072334bc819daab69b83db: Status 404 returned error can't find the container with id ce223dfbc93652ce2b90498e6c64ef01ecd747a6fe072334bc819daab69b83db
Sep 16 19:25:24.386337 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:24.386304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" event={"ID":"4a67cd12-6461-4916-900d-61fcfa222839","Type":"ContainerStarted","Data":"ce223dfbc93652ce2b90498e6c64ef01ecd747a6fe072334bc819daab69b83db"}
Sep 16 19:25:25.393661 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:25.393616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" event={"ID":"4a67cd12-6461-4916-900d-61fcfa222839","Type":"ContainerStarted","Data":"850a04accf66780084463aa32762f093a63aeea6d225a2dca40a230e78bcf966"}
Sep 16 19:25:25.394123 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:25.393768    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:25:25.417571 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:25.417507    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" podStartSLOduration=2.417489141 podStartE2EDuration="2.417489141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:25:25.415285708 +0000 UTC m=+2509.527082348" watchObservedRunningTime="2026-09-16 19:25:25.417489141 +0000 UTC m=+2509.529285786"
Sep 16 19:25:50.178166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:50.178124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:25:56.410028 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:25:56.409990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:26:00.578123 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:00.578077    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"]
Sep 16 19:26:00.588732 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:00.588695    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:00.663595 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:00.663558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:06.401626 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:06.401583    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:15.599051 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:15.598981    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main" containerID="cri-o://850a04accf66780084463aa32762f093a63aeea6d225a2dca40a230e78bcf966" gracePeriod=45
Sep 16 19:26:16.401201 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:16.401151    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:19.854501 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:19.854456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:19.972252 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:19.972205    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:20.073012 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:20.072973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:20.351130 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:20.351093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:20.693728 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:20.693680    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"]
Sep 16 19:26:20.743549 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:20.743514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:24.926708 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:24.926672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:26.401074 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:26.401016    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:26.403162 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:26.403139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:26:34.505700 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:34.505656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:35.719048 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:35.718982    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" podUID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" containerName="main" containerID="cri-o://c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52" gracePeriod=45
Sep 16 19:26:35.967143 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:35.967113    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:26:36.015845 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.015745    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tmp-dir\") pod \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") "
Sep 16 19:26:36.015845 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.015806    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpwqh\" (UniqueName: \"kubernetes.io/projected/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-kube-api-access-lpwqh\") pod \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") "
Sep 16 19:26:36.015845 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.015826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-dshm\") pod \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") "
Sep 16 19:26:36.016166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.015851    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-home\") pod \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") "
Sep 16 19:26:36.016166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016061    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-model-cache\") pod \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") "
Sep 16 19:26:36.016166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016116    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7754155e-56bf-48f6-b1bb-f5d1f83fbea1" (UID: "7754155e-56bf-48f6-b1bb-f5d1f83fbea1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:36.016166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016130    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tls-certs\") pod \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\" (UID: \"7754155e-56bf-48f6-b1bb-f5d1f83fbea1\") "
Sep 16 19:26:36.016166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-home" (OuterVolumeSpecName: "home") pod "7754155e-56bf-48f6-b1bb-f5d1f83fbea1" (UID: "7754155e-56bf-48f6-b1bb-f5d1f83fbea1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:36.016453 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016291    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-model-cache" (OuterVolumeSpecName: "model-cache") pod "7754155e-56bf-48f6-b1bb-f5d1f83fbea1" (UID: "7754155e-56bf-48f6-b1bb-f5d1f83fbea1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:36.016453 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016434    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.016636 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016453    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.016636 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.016466    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.018235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.018197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7754155e-56bf-48f6-b1bb-f5d1f83fbea1" (UID: "7754155e-56bf-48f6-b1bb-f5d1f83fbea1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:36.018354 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.018267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-kube-api-access-lpwqh" (OuterVolumeSpecName: "kube-api-access-lpwqh") pod "7754155e-56bf-48f6-b1bb-f5d1f83fbea1" (UID: "7754155e-56bf-48f6-b1bb-f5d1f83fbea1"). InnerVolumeSpecName "kube-api-access-lpwqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:36.018354 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.018275    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-dshm" (OuterVolumeSpecName: "dshm") pod "7754155e-56bf-48f6-b1bb-f5d1f83fbea1" (UID: "7754155e-56bf-48f6-b1bb-f5d1f83fbea1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:26:36.117023 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.116988    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.117023 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.117022    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpwqh\" (UniqueName: \"kubernetes.io/projected/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-kube-api-access-lpwqh\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.117305 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.117036    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7754155e-56bf-48f6-b1bb-f5d1f83fbea1-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:36.401201 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.401161    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:26:36.794178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.794083    2566 generic.go:356] "Generic (PLEG): container finished" podID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" containerID="c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52" exitCode=0
Sep 16 19:26:36.794178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.794143    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"
Sep 16 19:26:36.794178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.794159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" event={"ID":"7754155e-56bf-48f6-b1bb-f5d1f83fbea1","Type":"ContainerDied","Data":"c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52"}
Sep 16 19:26:36.794816 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.794236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" event={"ID":"7754155e-56bf-48f6-b1bb-f5d1f83fbea1","Type":"ContainerDied","Data":"18166b9704492092fc8923029b020825ab614bdabcc0bff18ce1b9a3527d33ff"}
Sep 16 19:26:36.794816 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.794259    2566 scope.go:117] "RemoveContainer" containerID="c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52"
Sep 16 19:26:36.803156 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.803137    2566 scope.go:117] "RemoveContainer" containerID="c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52"
Sep 16 19:26:36.803437 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:26:36.803420    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52\": container with ID starting with c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52 not found: ID does not exist" containerID="c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52"
Sep 16 19:26:36.803497 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.803445    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52"} err="failed to get container status \"c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52\": rpc error: code = NotFound desc = could not find container \"c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52\": container with ID starting with c04a2809ddc4006b4b684eade6b9ff6065c99ed83a73db10a362ebfd2c96da52 not found: ID does not exist"
Sep 16 19:26:36.829683 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.829642    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"]
Sep 16 19:26:36.835072 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.835042    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn"]
Sep 16 19:26:36.946179 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:36.946123    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-late-join-483adbac/late-v1-kserve-56cbc8456c-zk2bn" podUID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:26:38.403320 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:38.403273    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" path="/var/lib/kubelet/pods/7754155e-56bf-48f6-b1bb-f5d1f83fbea1/volumes"
Sep 16 19:26:41.737134 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:41.737093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:41.837972 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:41.837931    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:26:45.842608 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.842574    2566 generic.go:356] "Generic (PLEG): container finished" podID="4a67cd12-6461-4916-900d-61fcfa222839" containerID="850a04accf66780084463aa32762f093a63aeea6d225a2dca40a230e78bcf966" exitCode=0
Sep 16 19:26:45.842963 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.842687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" event={"ID":"4a67cd12-6461-4916-900d-61fcfa222839","Type":"ContainerDied","Data":"850a04accf66780084463aa32762f093a63aeea6d225a2dca40a230e78bcf966"}
Sep 16 19:26:45.842963 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.842740    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd" event={"ID":"4a67cd12-6461-4916-900d-61fcfa222839","Type":"ContainerDied","Data":"ce223dfbc93652ce2b90498e6c64ef01ecd747a6fe072334bc819daab69b83db"}
Sep 16 19:26:45.842963 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.842764    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce223dfbc93652ce2b90498e6c64ef01ecd747a6fe072334bc819daab69b83db"
Sep 16 19:26:45.856171 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.856141    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:26:45.909677 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.909634    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmzrt\" (UniqueName: \"kubernetes.io/projected/4a67cd12-6461-4916-900d-61fcfa222839-kube-api-access-vmzrt\") pod \"4a67cd12-6461-4916-900d-61fcfa222839\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") "
Sep 16 19:26:45.909901 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.909721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cd12-6461-4916-900d-61fcfa222839-tls-certs\") pod \"4a67cd12-6461-4916-900d-61fcfa222839\" (UID: \"4a67cd12-6461-4916-900d-61fcfa222839\") "
Sep 16 19:26:45.911989 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.911958    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a67cd12-6461-4916-900d-61fcfa222839-kube-api-access-vmzrt" (OuterVolumeSpecName: "kube-api-access-vmzrt") pod "4a67cd12-6461-4916-900d-61fcfa222839" (UID: "4a67cd12-6461-4916-900d-61fcfa222839"). InnerVolumeSpecName "kube-api-access-vmzrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:26:45.912080 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:45.912008    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a67cd12-6461-4916-900d-61fcfa222839-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a67cd12-6461-4916-900d-61fcfa222839" (UID: "4a67cd12-6461-4916-900d-61fcfa222839"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:26:46.010614 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:46.010561    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmzrt\" (UniqueName: \"kubernetes.io/projected/4a67cd12-6461-4916-900d-61fcfa222839-kube-api-access-vmzrt\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:46.010614 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:46.010610    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a67cd12-6461-4916-900d-61fcfa222839-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:26:46.847594 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:46.847503    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"
Sep 16 19:26:46.878629 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:46.878589    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"]
Sep 16 19:26:46.885473 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:46.885436    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-scheg7lfd"]
Sep 16 19:26:48.403061 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:48.403022    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a67cd12-6461-4916-900d-61fcfa222839" path="/var/lib/kubelet/pods/4a67cd12-6461-4916-900d-61fcfa222839/volumes"
Sep 16 19:26:58.868371 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:26:58.868272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:00.716142 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:00.716102    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:00.723239 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:00.723208    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"]
Sep 16 19:27:00.732109 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:27:00.732069    2566 log.go:32] "ExecSync cmd from runtime service failed" err=<
Sep 16 19:27:00.732109 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:27:00.732109 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 19:27:00.732109 ip-10-0-143-190 kubenswrapper[2566]:  > containerID="aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112" cmd=["/bin/sleep","15"]
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:00.732128    2566 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" message="" err=<
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]:  >
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:27:00.732140    2566 kuberuntime_container.go:755] "PreStop hook failed" err=<
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]:         , stdout: , stderr: , exit code -1
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]:  > pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" podUID="072ed10c-a128-437b-9f9b-4986bce70739" containerName="main" containerID="cri-o://aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"
Sep 16 19:27:00.732404 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:00.732192    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" podUID="072ed10c-a128-437b-9f9b-4986bce70739" containerName="main" containerID="cri-o://aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112" gracePeriod=60
Sep 16 19:27:01.079480 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.079441    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:01.086519 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.086495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:27:01.141932 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.141889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-dshm\") pod \"072ed10c-a128-437b-9f9b-4986bce70739\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") "
Sep 16 19:27:01.142141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.141987    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-model-cache\") pod \"072ed10c-a128-437b-9f9b-4986bce70739\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") "
Sep 16 19:27:01.142141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-home\") pod \"072ed10c-a128-437b-9f9b-4986bce70739\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") "
Sep 16 19:27:01.142141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142035    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6ljr\" (UniqueName: \"kubernetes.io/projected/072ed10c-a128-437b-9f9b-4986bce70739-kube-api-access-k6ljr\") pod \"072ed10c-a128-437b-9f9b-4986bce70739\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") "
Sep 16 19:27:01.142141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-tmp-dir\") pod \"072ed10c-a128-437b-9f9b-4986bce70739\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") "
Sep 16 19:27:01.142141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072ed10c-a128-437b-9f9b-4986bce70739-tls-certs\") pod \"072ed10c-a128-437b-9f9b-4986bce70739\" (UID: \"072ed10c-a128-437b-9f9b-4986bce70739\") "
Sep 16 19:27:01.142421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-model-cache" (OuterVolumeSpecName: "model-cache") pod "072ed10c-a128-437b-9f9b-4986bce70739" (UID: "072ed10c-a128-437b-9f9b-4986bce70739"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:01.142421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142306    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-home" (OuterVolumeSpecName: "home") pod "072ed10c-a128-437b-9f9b-4986bce70739" (UID: "072ed10c-a128-437b-9f9b-4986bce70739"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:01.142421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142375    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:01.142421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142391    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:01.142638 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.142599    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "072ed10c-a128-437b-9f9b-4986bce70739" (UID: "072ed10c-a128-437b-9f9b-4986bce70739"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:01.144313 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.144284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072ed10c-a128-437b-9f9b-4986bce70739-kube-api-access-k6ljr" (OuterVolumeSpecName: "kube-api-access-k6ljr") pod "072ed10c-a128-437b-9f9b-4986bce70739" (UID: "072ed10c-a128-437b-9f9b-4986bce70739"). InnerVolumeSpecName "kube-api-access-k6ljr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:01.144445 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.144393    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-dshm" (OuterVolumeSpecName: "dshm") pod "072ed10c-a128-437b-9f9b-4986bce70739" (UID: "072ed10c-a128-437b-9f9b-4986bce70739"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:01.144445 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.144419    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072ed10c-a128-437b-9f9b-4986bce70739-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "072ed10c-a128-437b-9f9b-4986bce70739" (UID: "072ed10c-a128-437b-9f9b-4986bce70739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:01.243643 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.243605    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6ljr\" (UniqueName: \"kubernetes.io/projected/072ed10c-a128-437b-9f9b-4986bce70739-kube-api-access-k6ljr\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:01.243643 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.243638    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:01.243643 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.243650    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072ed10c-a128-437b-9f9b-4986bce70739-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:01.243927 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.243658    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/072ed10c-a128-437b-9f9b-4986bce70739-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:27:01.925930 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.925858    2566 generic.go:356] "Generic (PLEG): container finished" podID="072ed10c-a128-437b-9f9b-4986bce70739" containerID="aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112" exitCode=0
Sep 16 19:27:01.925930 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.925912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" event={"ID":"072ed10c-a128-437b-9f9b-4986bce70739","Type":"ContainerDied","Data":"aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"}
Sep 16 19:27:01.926425 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.925966    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"
Sep 16 19:27:01.926425 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.925990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" event={"ID":"072ed10c-a128-437b-9f9b-4986bce70739","Type":"ContainerDied","Data":"698ecd34a1b1f1ca37362bb3c57ffb30f4b9fb5f2eed3f7a504b6b0c14b92b13"}
Sep 16 19:27:01.926425 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.926016    2566 scope.go:117] "RemoveContainer" containerID="aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"
Sep 16 19:27:01.936254 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.936224    2566 scope.go:117] "RemoveContainer" containerID="aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"
Sep 16 19:27:01.936607 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:27:01.936584    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112\": container with ID starting with aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112 not found: ID does not exist" containerID="aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"
Sep 16 19:27:01.936655 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.936618    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112"} err="failed to get container status \"aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112\": rpc error: code = NotFound desc = could not find container \"aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112\": container with ID starting with aaabecb27948cf7fa413907e868c7e0911cac5b7d1f6215003516caf8dd59112 not found: ID does not exist"
Sep 16 19:27:01.953353 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.953298    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"]
Sep 16 19:27:01.956172 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:01.956143    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp"]
Sep 16 19:27:02.042018 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:02.041970    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-84370a53/llmisvc-router-managed-test-llm-5b1e8f15-kserve-df5b5c977-5spvp" podUID="072ed10c-a128-437b-9f9b-4986bce70739" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded"
Sep 16 19:27:02.403039 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:02.402996    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072ed10c-a128-437b-9f9b-4986bce70739" path="/var/lib/kubelet/pods/072ed10c-a128-437b-9f9b-4986bce70739/volumes"
Sep 16 19:27:03.076514 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:03.076475    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:09.170549 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:09.170509    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:09.258456 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:09.258418    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:50.788711 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:50.788649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:50.883300 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:50.883260    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:52.912533 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:52.912493    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:27:57.819558 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:27:57.819521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:03.699066 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:03.699034    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:03.795986 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:03.795949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:05.633337 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.633285    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:05.852839 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.852803    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"]
Sep 16 19:28:05.853368 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853346    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" containerName="main"
Sep 16 19:28:05.853368 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853370    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853386    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072ed10c-a128-437b-9f9b-4986bce70739" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853396    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="072ed10c-a128-437b-9f9b-4986bce70739" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853426    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853436    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853519    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a67cd12-6461-4916-900d-61fcfa222839" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853529    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7754155e-56bf-48f6-b1bb-f5d1f83fbea1" containerName="main"
Sep 16 19:28:05.853570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.853541    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="072ed10c-a128-437b-9f9b-4986bce70739" containerName="main"
Sep 16 19:28:05.856951 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.856931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:05.861112 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.861087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"default-dockercfg-nj5jj\""
Sep 16 19:28:05.861235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.861165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"kube-root-ca.crt\""
Sep 16 19:28:05.861235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.861164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvca2d2d7d499abb359505529ebe02c136-kserve-self-signed-certs\""
Sep 16 19:28:05.861372 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.861165    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"openshift-service-ca.crt\""
Sep 16 19:28:05.861372 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.861178    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-router-managed-test-llm-4b931143-epp-sa-dockercfg-q4prr\""
Sep 16 19:28:05.868954 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.868930    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"]
Sep 16 19:28:05.886513 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:05.886479    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:06.025193 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.025158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a831a2a-5ce2-4671-84b8-7e2923d423ca-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.025398 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.025201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ckv\" (UniqueName: \"kubernetes.io/projected/8a831a2a-5ce2-4671-84b8-7e2923d423ca-kube-api-access-k2ckv\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.126231 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.126190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a831a2a-5ce2-4671-84b8-7e2923d423ca-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.126231 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.126230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ckv\" (UniqueName: \"kubernetes.io/projected/8a831a2a-5ce2-4671-84b8-7e2923d423ca-kube-api-access-k2ckv\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.128732 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.128705    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a831a2a-5ce2-4671-84b8-7e2923d423ca-tls-certs\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.135324 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.135299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ckv\" (UniqueName: \"kubernetes.io/projected/8a831a2a-5ce2-4671-84b8-7e2923d423ca-kube-api-access-k2ckv\") pod \"llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.168092 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.167999    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:06.443957 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:06.443766    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"]
Sep 16 19:28:06.445168 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:28:06.445138    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a831a2a_5ce2_4671_84b8_7e2923d423ca.slice/crio-9b8a9f8f609c5d5048397e86a859e35350650fd37c06399da6873f2f723f6e47 WatchSource:0}: Error finding container 9b8a9f8f609c5d5048397e86a859e35350650fd37c06399da6873f2f723f6e47: Status 404 returned error can't find the container with id 9b8a9f8f609c5d5048397e86a859e35350650fd37c06399da6873f2f723f6e47
Sep 16 19:28:07.273631 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:07.273589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" event={"ID":"8a831a2a-5ce2-4671-84b8-7e2923d423ca","Type":"ContainerStarted","Data":"ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce"}
Sep 16 19:28:07.274074 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:07.273649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" event={"ID":"8a831a2a-5ce2-4671-84b8-7e2923d423ca","Type":"ContainerStarted","Data":"9b8a9f8f609c5d5048397e86a859e35350650fd37c06399da6873f2f723f6e47"}
Sep 16 19:28:07.274074 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:07.273676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:07.295782 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:07.295730    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" podStartSLOduration=2.295712893 podStartE2EDuration="2.295712893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:07.295111145 +0000 UTC m=+2671.406907788" watchObservedRunningTime="2026-09-16 19:28:07.295712893 +0000 UTC m=+2671.407509535"
Sep 16 19:28:09.885963 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:09.885926    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:09.987455 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:09.987414    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:10.086586 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:10.086533    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:14.330721 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:14.330684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:14.392752 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:14.392708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:14.940312 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:14.940265    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:20.409348 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:20.409296    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:21.957971 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:21.957933    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:26.292357 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:26.292314    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:38.281112 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:38.281079    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:28:42.630971 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:42.630926    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:52.244382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:52.244322    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:57.984642 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:57.984596    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:28:58.086533 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:28:58.086489    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:45.279773 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.279733    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:45.435846 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.435808    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"]
Sep 16 19:29:45.439832 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.439790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.443366 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.443336    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisvc-model-qwen2-5-0-5b-rout-4f8c0978-epp-sa-dockercfg-tfdt4\""
Sep 16 19:29:45.443520 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.443398    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62fead0b\"/\"llmisv08544b88a8d963ffd553cc1f3ed82d16-kserve-self-signed-certs\""
Sep 16 19:29:45.453591 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.453541    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"]
Sep 16 19:29:45.473491 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.473444    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:29:45.553114 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.553014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cxsg\" (UniqueName: \"kubernetes.io/projected/03099dc0-5c76-4933-94b1-87b2fd615f9a-kube-api-access-8cxsg\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.553114 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.553075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03099dc0-5c76-4933-94b1-87b2fd615f9a-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.653931 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.653864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03099dc0-5c76-4933-94b1-87b2fd615f9a-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.654121 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.654004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cxsg\" (UniqueName: \"kubernetes.io/projected/03099dc0-5c76-4933-94b1-87b2fd615f9a-kube-api-access-8cxsg\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.656623 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.656594    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03099dc0-5c76-4933-94b1-87b2fd615f9a-tls-certs\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.664317 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.664286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cxsg\" (UniqueName: \"kubernetes.io/projected/03099dc0-5c76-4933-94b1-87b2fd615f9a-kube-api-access-8cxsg\") pod \"llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") " pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:45.754789 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:45.754745    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:46.030326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:46.030296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"]
Sep 16 19:29:46.032130 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:29:46.032092    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03099dc0_5c76_4933_94b1_87b2fd615f9a.slice/crio-a6b94fc700020fac29a2b64c48dac1a4aaf1f3fbbdc0b657907bf22c62e5d852 WatchSource:0}: Error finding container a6b94fc700020fac29a2b64c48dac1a4aaf1f3fbbdc0b657907bf22c62e5d852: Status 404 returned error can't find the container with id a6b94fc700020fac29a2b64c48dac1a4aaf1f3fbbdc0b657907bf22c62e5d852
Sep 16 19:29:46.807677 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:46.807634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" event={"ID":"03099dc0-5c76-4933-94b1-87b2fd615f9a","Type":"ContainerStarted","Data":"930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3"}
Sep 16 19:29:46.808115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:46.807698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" event={"ID":"03099dc0-5c76-4933-94b1-87b2fd615f9a","Type":"ContainerStarted","Data":"a6b94fc700020fac29a2b64c48dac1a4aaf1f3fbbdc0b657907bf22c62e5d852"}
Sep 16 19:29:46.808115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:46.807840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:29:46.826217 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:46.826151    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" podStartSLOduration=1.826129071 podStartE2EDuration="1.826129071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:46.825843181 +0000 UTC m=+2770.937639849" watchObservedRunningTime="2026-09-16 19:29:46.826129071 +0000 UTC m=+2770.937925716"
Sep 16 19:29:56.460022 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:29:56.459986    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:17.814641 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:17.814551    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:30:19.764062 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:19.763790    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"]
Sep 16 19:30:19.764790 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:19.764761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:19.841582 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:19.841528    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:27.814787 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:27.814747    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:34.787765 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:34.787710    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main" containerID="cri-o://930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3" gracePeriod=45
Sep 16 19:30:37.814289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:37.814244    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:44.199432 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:44.199390    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:30:47.814539 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:47.814495    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:30:47.814957 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:47.814617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:30:57.814382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:30:57.814338    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:05.029366 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.029337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:31:05.157831 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.157793    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cxsg\" (UniqueName: \"kubernetes.io/projected/03099dc0-5c76-4933-94b1-87b2fd615f9a-kube-api-access-8cxsg\") pod \"03099dc0-5c76-4933-94b1-87b2fd615f9a\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") "
Sep 16 19:31:05.158027 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.157869    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03099dc0-5c76-4933-94b1-87b2fd615f9a-tls-certs\") pod \"03099dc0-5c76-4933-94b1-87b2fd615f9a\" (UID: \"03099dc0-5c76-4933-94b1-87b2fd615f9a\") "
Sep 16 19:31:05.160083 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.160045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03099dc0-5c76-4933-94b1-87b2fd615f9a-kube-api-access-8cxsg" (OuterVolumeSpecName: "kube-api-access-8cxsg") pod "03099dc0-5c76-4933-94b1-87b2fd615f9a" (UID: "03099dc0-5c76-4933-94b1-87b2fd615f9a"). InnerVolumeSpecName "kube-api-access-8cxsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:05.160196 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.160101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03099dc0-5c76-4933-94b1-87b2fd615f9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03099dc0-5c76-4933-94b1-87b2fd615f9a" (UID: "03099dc0-5c76-4933-94b1-87b2fd615f9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:05.224120 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.224079    2566 generic.go:356] "Generic (PLEG): container finished" podID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerID="930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3" exitCode=0
Sep 16 19:31:05.224294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.224133    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"
Sep 16 19:31:05.224294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.224180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" event={"ID":"03099dc0-5c76-4933-94b1-87b2fd615f9a","Type":"ContainerDied","Data":"930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3"}
Sep 16 19:31:05.224294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.224244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g" event={"ID":"03099dc0-5c76-4933-94b1-87b2fd615f9a","Type":"ContainerDied","Data":"a6b94fc700020fac29a2b64c48dac1a4aaf1f3fbbdc0b657907bf22c62e5d852"}
Sep 16 19:31:05.224294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.224274    2566 scope.go:117] "RemoveContainer" containerID="930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3"
Sep 16 19:31:05.233282 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.233254    2566 scope.go:117] "RemoveContainer" containerID="930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3"
Sep 16 19:31:05.233634 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:31:05.233605    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3\": container with ID starting with 930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3 not found: ID does not exist" containerID="930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3"
Sep 16 19:31:05.233702 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.233643    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3"} err="failed to get container status \"930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3\": rpc error: code = NotFound desc = could not find container \"930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3\": container with ID starting with 930e909b535689eb1f50207a7530a300b479901e573ea9dd07be6905b0ed1da3 not found: ID does not exist"
Sep 16 19:31:05.249525 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.249493    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"]
Sep 16 19:31:05.251663 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.251640    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvc08544b88a8d963ffd553cc1f3ed82d16-kserve-router-schemxj5g"]
Sep 16 19:31:05.259326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.259294    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03099dc0-5c76-4933-94b1-87b2fd615f9a-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:05.259326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.259322    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cxsg\" (UniqueName: \"kubernetes.io/projected/03099dc0-5c76-4933-94b1-87b2fd615f9a-kube-api-access-8cxsg\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:31:05.615289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.615199    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:05.795029 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.794985    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:05.958167 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:05.958124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:06.403547 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:06.403509    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" path="/var/lib/kubelet/pods/03099dc0-5c76-4933-94b1-87b2fd615f9a/volumes"
Sep 16 19:31:06.578115 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:06.578079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:10.655785 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:10.655734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:14.127529 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:14.127490    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:36.770189 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:36.770104    2566 scope.go:117] "RemoveContainer" containerID="850a04accf66780084463aa32762f093a63aeea6d225a2dca40a230e78bcf966"
Sep 16 19:31:37.306332 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:37.306290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:38.666046 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:38.666010    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:38.666750 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:38.666721    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"]
Sep 16 19:31:38.762872 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:38.762831    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:48.282148 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:48.282106    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:48.584370 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:48.584272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:53.686072 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:53.685997    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main" containerID="cri-o://ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce" gracePeriod=45
Sep 16 19:31:55.237003 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:55.236967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:55.379699 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:55.379657    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:58.281659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:58.281615    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:31:59.212553 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:59.212513    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:31:59.314867 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:31:59.314825    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:01.609040 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:01.608997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:08.281977 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:08.281931    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:08.282515 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:08.282050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:32:18.281890 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:18.281843    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:32:23.940812 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:23.940785    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:32:24.045827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.045732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a831a2a-5ce2-4671-84b8-7e2923d423ca-tls-certs\") pod \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") "
Sep 16 19:32:24.045827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.045783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2ckv\" (UniqueName: \"kubernetes.io/projected/8a831a2a-5ce2-4671-84b8-7e2923d423ca-kube-api-access-k2ckv\") pod \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\" (UID: \"8a831a2a-5ce2-4671-84b8-7e2923d423ca\") "
Sep 16 19:32:24.047915 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.047866    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a831a2a-5ce2-4671-84b8-7e2923d423ca-kube-api-access-k2ckv" (OuterVolumeSpecName: "kube-api-access-k2ckv") pod "8a831a2a-5ce2-4671-84b8-7e2923d423ca" (UID: "8a831a2a-5ce2-4671-84b8-7e2923d423ca"). InnerVolumeSpecName "kube-api-access-k2ckv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:24.047994 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.047935    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a831a2a-5ce2-4671-84b8-7e2923d423ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a831a2a-5ce2-4671-84b8-7e2923d423ca" (UID: "8a831a2a-5ce2-4671-84b8-7e2923d423ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:24.147388 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.147348    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a831a2a-5ce2-4671-84b8-7e2923d423ca-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:24.147388 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.147382    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ckv\" (UniqueName: \"kubernetes.io/projected/8a831a2a-5ce2-4671-84b8-7e2923d423ca-kube-api-access-k2ckv\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:32:24.641074 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.641038    2566 generic.go:356] "Generic (PLEG): container finished" podID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerID="ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce" exitCode=0
Sep 16 19:32:24.641268 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.641100    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"
Sep 16 19:32:24.641268 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.641101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" event={"ID":"8a831a2a-5ce2-4671-84b8-7e2923d423ca","Type":"ContainerDied","Data":"ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce"}
Sep 16 19:32:24.641268 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.641253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4" event={"ID":"8a831a2a-5ce2-4671-84b8-7e2923d423ca","Type":"ContainerDied","Data":"9b8a9f8f609c5d5048397e86a859e35350650fd37c06399da6873f2f723f6e47"}
Sep 16 19:32:24.641428 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.641287    2566 scope.go:117] "RemoveContainer" containerID="ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce"
Sep 16 19:32:24.650330 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.650309    2566 scope.go:117] "RemoveContainer" containerID="ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce"
Sep 16 19:32:24.650631 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:32:24.650610    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce\": container with ID starting with ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce not found: ID does not exist" containerID="ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce"
Sep 16 19:32:24.650689 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.650639    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce"} err="failed to get container status \"ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce\": rpc error: code = NotFound desc = could not find container \"ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce\": container with ID starting with ed289c1f51b0a2d29a9549e9967d093da961ddf68869ef29ed2ad8f3718e87ce not found: ID does not exist"
Sep 16 19:32:24.660706 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.660615    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"]
Sep 16 19:32:24.662973 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:24.662945    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62fead0b/llmisvcca2d2d7d499abb359505529ebe02c136-kserve-router-sche9tjm4"]
Sep 16 19:32:26.403520 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:26.403487    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" path="/var/lib/kubelet/pods/8a831a2a-5ce2-4671-84b8-7e2923d423ca/volumes"
Sep 16 19:32:29.595137 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:29.595098    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:29.734916 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:29.734654    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:36.400195 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:36.400156    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:42.230110 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:42.230055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:42.305403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:42.305355    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:32:52.776102 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:32:52.776058    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:12.150229 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.150135    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:12.376541 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.376507    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"]
Sep 16 19:33:12.376908 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.376874    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main"
Sep 16 19:33:12.376908 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.376909    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main"
Sep 16 19:33:12.377054 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.376924    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main"
Sep 16 19:33:12.377054 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.376930    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main"
Sep 16 19:33:12.377054 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.376996    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a831a2a-5ce2-4671-84b8-7e2923d423ca" containerName="main"
Sep 16 19:33:12.377054 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.377006    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03099dc0-5c76-4933-94b1-87b2fd615f9a" containerName="main"
Sep 16 19:33:12.380559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.380536    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.383365 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.383334    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"kube-root-ca.crt\""
Sep 16 19:33:12.384829 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.384809    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"openshift-service-ca.crt\""
Sep 16 19:33:12.384829 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.384824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"default-dockercfg-5vv2l\""
Sep 16 19:33:12.385026 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.384912    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Sep 16 19:33:12.385026 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.384935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-5bc7dccd\"/\"llmisvc-model-fb-opt-125m-with-7ca60146-epp-sa-dockercfg-ff4k2\""
Sep 16 19:33:12.402695 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.402619    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"]
Sep 16 19:33:12.431031 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.430996    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:12.449703 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.449666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc91e667-9750-4a54-8d89-8231d047801f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.449865 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.449721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v74m4\" (UniqueName: \"kubernetes.io/projected/dc91e667-9750-4a54-8d89-8231d047801f-kube-api-access-v74m4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.550303 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.550270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc91e667-9750-4a54-8d89-8231d047801f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.550472 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.550324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v74m4\" (UniqueName: \"kubernetes.io/projected/dc91e667-9750-4a54-8d89-8231d047801f-kube-api-access-v74m4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.552865 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.552831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc91e667-9750-4a54-8d89-8231d047801f-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.559676 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.559638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v74m4\" (UniqueName: \"kubernetes.io/projected/dc91e667-9750-4a54-8d89-8231d047801f-kube-api-access-v74m4\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") " pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.691976 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.691853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:12.990254 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:12.990227    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"]
Sep 16 19:33:12.992106 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:33:12.992074    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc91e667_9750_4a54_8d89_8231d047801f.slice/crio-99bd80c0e30078a633d893e008f25835443734d4e2b5a29d396d49b96e078e31 WatchSource:0}: Error finding container 99bd80c0e30078a633d893e008f25835443734d4e2b5a29d396d49b96e078e31: Status 404 returned error can't find the container with id 99bd80c0e30078a633d893e008f25835443734d4e2b5a29d396d49b96e078e31
Sep 16 19:33:13.900171 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:13.900135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" event={"ID":"dc91e667-9750-4a54-8d89-8231d047801f","Type":"ContainerStarted","Data":"e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760"}
Sep 16 19:33:13.900692 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:13.900223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" event={"ID":"dc91e667-9750-4a54-8d89-8231d047801f","Type":"ContainerStarted","Data":"99bd80c0e30078a633d893e008f25835443734d4e2b5a29d396d49b96e078e31"}
Sep 16 19:33:13.900692 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:13.900253    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:13.931944 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:13.931863    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podStartSLOduration=1.9318405749999998 podStartE2EDuration="1.931840575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:13.928996767 +0000 UTC m=+2978.040793408" watchObservedRunningTime="2026-09-16 19:33:13.931840575 +0000 UTC m=+2978.043637218"
Sep 16 19:33:24.271368 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:24.271306    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:27.597303 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:27.597257    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:44.908042 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:44.908006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:33:50.256794 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:50.256753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:50.354535 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:50.354493    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:50.587027 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:50.586991    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:50.688725 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:50.688693    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:54.145932 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:54.145892    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:33:55.332719 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:33:55.332673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:14.391231 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:34:14.391194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:24.219280 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:34:24.219229    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:32.765032 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:34:32.764927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:34:58.982571 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:34:58.982532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:03.919466 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:03.919429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:04.014258 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:04.014216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:08.960419 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:08.960373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:12.025041 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.025000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:12.582810 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.582771    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"]
Sep 16 19:35:12.586263 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.586240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.590039 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.590013    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisvc-model-fb-opt-125m-route-87882a8e-epp-sa-dockercfg-2brgl\""
Sep 16 19:35:12.590187 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.590091    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"default-dockercfg-264n7\""
Sep 16 19:35:12.590187 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.590099    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"kube-root-ca.crt\""
Sep 16 19:35:12.590187 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.590134    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"openshift-service-ca.crt\""
Sep 16 19:35:12.590349 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.590289    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-079cb970\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Sep 16 19:35:12.601235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.601212    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"]
Sep 16 19:35:12.617204 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.617168    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:12.685302 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.685252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48886a04-03b9-4c01-8d05-a33a149fb966-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.685302 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.685300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgh2z\" (UniqueName: \"kubernetes.io/projected/48886a04-03b9-4c01-8d05-a33a149fb966-kube-api-access-lgh2z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.786273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.786231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48886a04-03b9-4c01-8d05-a33a149fb966-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.786273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.786272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgh2z\" (UniqueName: \"kubernetes.io/projected/48886a04-03b9-4c01-8d05-a33a149fb966-kube-api-access-lgh2z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.788785 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.788757    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48886a04-03b9-4c01-8d05-a33a149fb966-tls-certs\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.795101 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.795066    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgh2z\" (UniqueName: \"kubernetes.io/projected/48886a04-03b9-4c01-8d05-a33a149fb966-kube-api-access-lgh2z\") pod \"llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") " pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:12.897092 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:12.897051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:13.173528 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:13.173500    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"]
Sep 16 19:35:13.177703 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:35:13.177662    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48886a04_03b9_4c01_8d05_a33a149fb966.slice/crio-9123d5d05282f33b56a7e3418ed24531fd62379efcf15d92933b5ac63512f560 WatchSource:0}: Error finding container 9123d5d05282f33b56a7e3418ed24531fd62379efcf15d92933b5ac63512f560: Status 404 returned error can't find the container with id 9123d5d05282f33b56a7e3418ed24531fd62379efcf15d92933b5ac63512f560
Sep 16 19:35:13.531909 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:13.531793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" event={"ID":"48886a04-03b9-4c01-8d05-a33a149fb966","Type":"ContainerStarted","Data":"905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf"}
Sep 16 19:35:13.531909 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:13.531851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" event={"ID":"48886a04-03b9-4c01-8d05-a33a149fb966","Type":"ContainerStarted","Data":"9123d5d05282f33b56a7e3418ed24531fd62379efcf15d92933b5ac63512f560"}
Sep 16 19:35:13.532128 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:13.531976    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:13.552716 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:13.552651    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" podStartSLOduration=1.552633913 podStartE2EDuration="1.552633913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:13.548009083 +0000 UTC m=+3097.659805723" watchObservedRunningTime="2026-09-16 19:35:13.552633913 +0000 UTC m=+3097.664430553"
Sep 16 19:35:19.464168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:19.464124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:20.574001 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:20.573961    2566 generic.go:356] "Generic (PLEG): container finished" podID="78a6b5a3-3645-4693-a957-3354e0c592ca" containerID="12f233f9c1555153b503ec3cfdf855b2d08cb228d0263edece7494fc2d109b4b" exitCode=2
Sep 16 19:35:20.574506 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:20.574021    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm" event={"ID":"78a6b5a3-3645-4693-a957-3354e0c592ca","Type":"ContainerDied","Data":"12f233f9c1555153b503ec3cfdf855b2d08cb228d0263edece7494fc2d109b4b"}
Sep 16 19:35:20.574506 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:20.574478    2566 scope.go:117] "RemoveContainer" containerID="12f233f9c1555153b503ec3cfdf855b2d08cb228d0263edece7494fc2d109b4b"
Sep 16 19:35:20.575471 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:20.575440    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:35:20.575573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:20.575520    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 16 19:35:20.703097 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:20.703060    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 19:35:21.582760 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:21.582719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm" event={"ID":"78a6b5a3-3645-4693-a957-3354e0c592ca","Type":"ContainerStarted","Data":"0d641f0ddc4d7f9b42581f9f581c2c6f77d0a5395d17c0e0331833a5132404e7"}
Sep 16 19:35:21.583301 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:21.582951    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 19:35:21.583567 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:21.583548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-mkxxm"
Sep 16 19:35:22.546918 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:22.546860    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:22.557616 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:22.557042    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"]
Sep 16 19:35:22.639734 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:22.639693    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:24.907457 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:24.907415    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:29.658433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:29.658396    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:29.758456 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:29.758419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:30.662525 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:30.662479    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:31.284653 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:31.284615    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:34.907422 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:34.907377    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:37.066642 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:37.066594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:37.578169 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:37.578117    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main" containerID="cri-o://e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760" gracePeriod=45
Sep 16 19:35:44.539667 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:44.539630    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:35:44.908232 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:44.908188    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:35:44.908430 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:44.908301    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:35:46.945010 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:46.944973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:51.924346 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:51.924308    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:52.034237 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:52.034200    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:35:54.907938 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:35:54.907870    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:04.907521 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:04.907431    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:36:07.829524 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.829494    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:36:07.836075 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.836007    2566 generic.go:356] "Generic (PLEG): container finished" podID="dc91e667-9750-4a54-8d89-8231d047801f" containerID="e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760" exitCode=0
Sep 16 19:36:07.836075 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.836045    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"
Sep 16 19:36:07.836075 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.836056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" event={"ID":"dc91e667-9750-4a54-8d89-8231d047801f","Type":"ContainerDied","Data":"e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760"}
Sep 16 19:36:07.836278 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.836102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf" event={"ID":"dc91e667-9750-4a54-8d89-8231d047801f","Type":"ContainerDied","Data":"99bd80c0e30078a633d893e008f25835443734d4e2b5a29d396d49b96e078e31"}
Sep 16 19:36:07.836278 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.836125    2566 scope.go:117] "RemoveContainer" containerID="e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760"
Sep 16 19:36:07.844649 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.844629    2566 scope.go:117] "RemoveContainer" containerID="e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760"
Sep 16 19:36:07.844927 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:36:07.844900    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760\": container with ID starting with e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760 not found: ID does not exist" containerID="e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760"
Sep 16 19:36:07.845027 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.844934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760"} err="failed to get container status \"e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760\": rpc error: code = NotFound desc = could not find container \"e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760\": container with ID starting with e39d47e6e8ee0e8ed3aa726f1196e60ad90e6c0f42dbb82c0a97d08d93137760 not found: ID does not exist"
Sep 16 19:36:07.974178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.974138    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc91e667-9750-4a54-8d89-8231d047801f-tls-certs\") pod \"dc91e667-9750-4a54-8d89-8231d047801f\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") "
Sep 16 19:36:07.974178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.974183    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v74m4\" (UniqueName: \"kubernetes.io/projected/dc91e667-9750-4a54-8d89-8231d047801f-kube-api-access-v74m4\") pod \"dc91e667-9750-4a54-8d89-8231d047801f\" (UID: \"dc91e667-9750-4a54-8d89-8231d047801f\") "
Sep 16 19:36:07.976494 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.976459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc91e667-9750-4a54-8d89-8231d047801f-kube-api-access-v74m4" (OuterVolumeSpecName: "kube-api-access-v74m4") pod "dc91e667-9750-4a54-8d89-8231d047801f" (UID: "dc91e667-9750-4a54-8d89-8231d047801f"). InnerVolumeSpecName "kube-api-access-v74m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:36:07.976494 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:07.976469    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc91e667-9750-4a54-8d89-8231d047801f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dc91e667-9750-4a54-8d89-8231d047801f" (UID: "dc91e667-9750-4a54-8d89-8231d047801f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:36:08.075273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:08.075222    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc91e667-9750-4a54-8d89-8231d047801f-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:36:08.075273 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:08.075269    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v74m4\" (UniqueName: \"kubernetes.io/projected/dc91e667-9750-4a54-8d89-8231d047801f-kube-api-access-v74m4\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:36:08.159460 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:08.159422    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"]
Sep 16 19:36:08.164675 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:08.164640    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-5bc7dccd/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-router-schec42vf"]
Sep 16 19:36:08.409329 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:08.409239    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc91e667-9750-4a54-8d89-8231d047801f" path="/var/lib/kubelet/pods/dc91e667-9750-4a54-8d89-8231d047801f/volumes"
Sep 16 19:36:11.465354 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:11.465318    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:11.551731 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:11.551684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:14.762741 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.762701    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:14.973766 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.973720    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"]
Sep 16 19:36:14.974325 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.974303    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main"
Sep 16 19:36:14.974412 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.974328    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main"
Sep 16 19:36:14.974473 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.974437    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc91e667-9750-4a54-8d89-8231d047801f" containerName="main"
Sep 16 19:36:14.979352 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.979325    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:14.982245 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.982221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"openshift-service-ca.crt\""
Sep 16 19:36:14.983310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.983287    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisvc-model-fb-opt-125m-with-ba4d693a-epp-sa-dockercfg-m8pff\""
Sep 16 19:36:14.983464 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.983451    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"kube-root-ca.crt\""
Sep 16 19:36:14.983540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.983489    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"default-dockercfg-gldcd\""
Sep 16 19:36:14.983540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.983502    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-733055c7\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Sep 16 19:36:14.989441 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:14.989403    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"]
Sep 16 19:36:15.023427 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.023334    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:15.028071 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.028043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.028245 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.028081    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpglj\" (UniqueName: \"kubernetes.io/projected/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-kube-api-access-zpglj\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.129149 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.129102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.129149 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.129153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpglj\" (UniqueName: \"kubernetes.io/projected/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-kube-api-access-zpglj\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.131642 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.131613    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.139307 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.139275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpglj\" (UniqueName: \"kubernetes.io/projected/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-kube-api-access-zpglj\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") " pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.291157 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.291068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.586821 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.586785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"]
Sep 16 19:36:15.589150 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:36:15.589116    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec2499f3_d0b1_4b69_ba23_e1da345cf03d.slice/crio-6387fe75ac985feca53eb9d3c99d12eb2cdba669e27d63b2a8dfe5940fcc93c2 WatchSource:0}: Error finding container 6387fe75ac985feca53eb9d3c99d12eb2cdba669e27d63b2a8dfe5940fcc93c2: Status 404 returned error can't find the container with id 6387fe75ac985feca53eb9d3c99d12eb2cdba669e27d63b2a8dfe5940fcc93c2
Sep 16 19:36:15.881688 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.881583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" event={"ID":"ec2499f3-d0b1-4b69-ba23-e1da345cf03d","Type":"ContainerStarted","Data":"8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3"}
Sep 16 19:36:15.881688 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.881646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" event={"ID":"ec2499f3-d0b1-4b69-ba23-e1da345cf03d","Type":"ContainerStarted","Data":"6387fe75ac985feca53eb9d3c99d12eb2cdba669e27d63b2a8dfe5940fcc93c2"}
Sep 16 19:36:15.881688 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.881675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:36:15.903067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:15.903000    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podStartSLOduration=1.902982704 podStartE2EDuration="1.902982704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:15.90141892 +0000 UTC m=+3160.013215563" watchObservedRunningTime="2026-09-16 19:36:15.902982704 +0000 UTC m=+3160.014779345"
Sep 16 19:36:18.069058 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:18.069020    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:23.473261 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:23.473225    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:24.394216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:24.394174    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:24.495568 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:24.495529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:36:46.889840 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:36:46.889804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:37:17.657095 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:17.657055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:17.674756 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:17.674724    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"]
Sep 16 19:37:17.911215 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:17.911126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:24.539654 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:24.539612    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:32.694596 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:32.694502    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main" containerID="cri-o://905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf" gracePeriod=45
Sep 16 19:37:34.539986 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:34.539932    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:36.724343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:36.724302    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"]
Sep 16 19:37:36.733256 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:36.733220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:36.818201 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:36.818166    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:37:36.889486 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:36.889445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:44.539764 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:44.539725    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:44.540191 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:44.539856    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:37:46.889455 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:46.889412    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:51.744747 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:51.744685    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main" containerID="cri-o://8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3" gracePeriod=45
Sep 16 19:37:54.540560 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:54.540518    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:56.890033 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:56.889993    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:37:56.890507 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:37:56.890098    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:38:02.951766 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:02.951734    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:38:02.974196 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:02.974112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48886a04-03b9-4c01-8d05-a33a149fb966-tls-certs\") pod \"48886a04-03b9-4c01-8d05-a33a149fb966\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") "
Sep 16 19:38:02.974343 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:02.974208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgh2z\" (UniqueName: \"kubernetes.io/projected/48886a04-03b9-4c01-8d05-a33a149fb966-kube-api-access-lgh2z\") pod \"48886a04-03b9-4c01-8d05-a33a149fb966\" (UID: \"48886a04-03b9-4c01-8d05-a33a149fb966\") "
Sep 16 19:38:02.976564 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:02.976526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48886a04-03b9-4c01-8d05-a33a149fb966-kube-api-access-lgh2z" (OuterVolumeSpecName: "kube-api-access-lgh2z") pod "48886a04-03b9-4c01-8d05-a33a149fb966" (UID: "48886a04-03b9-4c01-8d05-a33a149fb966"). InnerVolumeSpecName "kube-api-access-lgh2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:02.976729 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:02.976671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48886a04-03b9-4c01-8d05-a33a149fb966-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48886a04-03b9-4c01-8d05-a33a149fb966" (UID: "48886a04-03b9-4c01-8d05-a33a149fb966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:03.075321 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.075264    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgh2z\" (UniqueName: \"kubernetes.io/projected/48886a04-03b9-4c01-8d05-a33a149fb966-kube-api-access-lgh2z\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:38:03.075321 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.075314    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48886a04-03b9-4c01-8d05-a33a149fb966-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:38:03.442174 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.442136    2566 generic.go:356] "Generic (PLEG): container finished" podID="48886a04-03b9-4c01-8d05-a33a149fb966" containerID="905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf" exitCode=0
Sep 16 19:38:03.442382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.442187    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"
Sep 16 19:38:03.442382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.442203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" event={"ID":"48886a04-03b9-4c01-8d05-a33a149fb966","Type":"ContainerDied","Data":"905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf"}
Sep 16 19:38:03.442382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.442257    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb" event={"ID":"48886a04-03b9-4c01-8d05-a33a149fb966","Type":"ContainerDied","Data":"9123d5d05282f33b56a7e3418ed24531fd62379efcf15d92933b5ac63512f560"}
Sep 16 19:38:03.442382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.442278    2566 scope.go:117] "RemoveContainer" containerID="905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf"
Sep 16 19:38:03.451387 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.451364    2566 scope.go:117] "RemoveContainer" containerID="905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf"
Sep 16 19:38:03.451654 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:38:03.451625    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf\": container with ID starting with 905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf not found: ID does not exist" containerID="905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf"
Sep 16 19:38:03.451732 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.451665    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf"} err="failed to get container status \"905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf\": rpc error: code = NotFound desc = could not find container \"905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf\": container with ID starting with 905f4727ce595240432f509bec9cb2da5e857a2df0a1aff87ddc823497aab5bf not found: ID does not exist"
Sep 16 19:38:03.472506 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.472463    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"]
Sep 16 19:38:03.481356 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:03.481318    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-079cb970/llmisvca690bbc929faec8bc98c767f16c003c1-kserve-router-schewhgjb"]
Sep 16 19:38:04.403027 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:04.402990    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" path="/var/lib/kubelet/pods/48886a04-03b9-4c01-8d05-a33a149fb966/volumes"
Sep 16 19:38:06.889741 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:06.889701    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:08.264413 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:08.264376    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:08.364469 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:08.364430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:12.537295 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:12.537243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:14.174353 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:14.174317    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:16.889611 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:16.889570    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:38:19.909958 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:19.909915    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:20.014397 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:20.014316    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:22.007329 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.007262    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:38:22.145192 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.145157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpglj\" (UniqueName: \"kubernetes.io/projected/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-kube-api-access-zpglj\") pod \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") "
Sep 16 19:38:22.145384 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.145235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-tls-certs\") pod \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\" (UID: \"ec2499f3-d0b1-4b69-ba23-e1da345cf03d\") "
Sep 16 19:38:22.147434 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.147407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-kube-api-access-zpglj" (OuterVolumeSpecName: "kube-api-access-zpglj") pod "ec2499f3-d0b1-4b69-ba23-e1da345cf03d" (UID: "ec2499f3-d0b1-4b69-ba23-e1da345cf03d"). InnerVolumeSpecName "kube-api-access-zpglj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:38:22.147548 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.147445    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ec2499f3-d0b1-4b69-ba23-e1da345cf03d" (UID: "ec2499f3-d0b1-4b69-ba23-e1da345cf03d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:38:22.245817 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.245776    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpglj\" (UniqueName: \"kubernetes.io/projected/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-kube-api-access-zpglj\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:38:22.245817 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.245811    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec2499f3-d0b1-4b69-ba23-e1da345cf03d-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:38:22.544543 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.544505    2566 generic.go:356] "Generic (PLEG): container finished" podID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerID="8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3" exitCode=0
Sep 16 19:38:22.544543 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.544552    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"
Sep 16 19:38:22.544827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.544616    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" event={"ID":"ec2499f3-d0b1-4b69-ba23-e1da345cf03d","Type":"ContainerDied","Data":"8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3"}
Sep 16 19:38:22.544827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.544685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w" event={"ID":"ec2499f3-d0b1-4b69-ba23-e1da345cf03d","Type":"ContainerDied","Data":"6387fe75ac985feca53eb9d3c99d12eb2cdba669e27d63b2a8dfe5940fcc93c2"}
Sep 16 19:38:22.544827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.544716    2566 scope.go:117] "RemoveContainer" containerID="8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3"
Sep 16 19:38:22.554779 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.554755    2566 scope.go:117] "RemoveContainer" containerID="8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3"
Sep 16 19:38:22.555217 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:38:22.555187    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3\": container with ID starting with 8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3 not found: ID does not exist" containerID="8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3"
Sep 16 19:38:22.555325 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.555224    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3"} err="failed to get container status \"8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3\": rpc error: code = NotFound desc = could not find container \"8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3\": container with ID starting with 8c334470ebc3da45c16b63c6f6c2bbbac0d9da5b45a6eb4c8f7a61f5fed8cec3 not found: ID does not exist"
Sep 16 19:38:22.563511 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.563484    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"]
Sep 16 19:38:22.568210 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:22.568181    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-733055c7/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-router-schezx45w"]
Sep 16 19:38:24.402416 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:24.402380    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" path="/var/lib/kubelet/pods/ec2499f3-d0b1-4b69-ba23-e1da345cf03d/volumes"
Sep 16 19:38:26.743373 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:26.743335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:26.848743 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:26.848707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:26.979602 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:26.979566    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:29.054202 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:29.054166    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:29.154589 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:29.154546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:29.265047 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:29.265011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:33.401329 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:33.401286    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:34.848719 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.848674    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"]
Sep 16 19:38:34.849244 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.849229    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main"
Sep 16 19:38:34.849403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.849251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main"
Sep 16 19:38:34.849403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.849274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main"
Sep 16 19:38:34.849403 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.849282    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main"
Sep 16 19:38:34.849573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.849402    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec2499f3-d0b1-4b69-ba23-e1da345cf03d" containerName="main"
Sep 16 19:38:34.849573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.849419    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="48886a04-03b9-4c01-8d05-a33a149fb966" containerName="main"
Sep 16 19:38:34.852084 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.852051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:34.854400 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.854373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.857900 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.857862    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"default-dockercfg-bx87j\""
Sep 16 19:38:34.858733 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.858709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"llmisv20d1e8ef181cac7effb0fab7b8590b15-kserve-self-signed-certs\""
Sep 16 19:38:34.859767 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.859748    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"kube-root-ca.crt\""
Sep 16 19:38:34.860202 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.860185    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-7bc4bd10\"/\"openshift-service-ca.crt\""
Sep 16 19:38:34.872178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.872152    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"]
Sep 16 19:38:34.958402 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.958402 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958399    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.958659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.958659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl5d9\" (UniqueName: \"kubernetes.io/projected/786faea0-23df-4f14-9db8-42066b13ed76-kube-api-access-zl5d9\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.958659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.958659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:34.958659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:34.958640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786faea0-23df-4f14-9db8-42066b13ed76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059665 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786faea0-23df-4f14-9db8-42066b13ed76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl5d9\" (UniqueName: \"kubernetes.io/projected/786faea0-23df-4f14-9db8-42066b13ed76-kube-api-access-zl5d9\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.059863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.059822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.060304 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.060136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-home\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.060304 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.060161    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.060304 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.060275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-model-cache\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.060304 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.060295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.062355 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.062303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-dshm\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.062666 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.062645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786faea0-23df-4f14-9db8-42066b13ed76-tls-certs\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.070592 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.070565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl5d9\" (UniqueName: \"kubernetes.io/projected/786faea0-23df-4f14-9db8-42066b13ed76-kube-api-access-zl5d9\") pod \"llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") " pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.166396 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.166349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:38:35.210955 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.210908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:35.451848 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.451817    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"]
Sep 16 19:38:35.500582 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.500560    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:38:35.615593 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:35.615555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerStarted","Data":"05ab445ef193b29e0ac1a0849d8b67c88e820c8004570f66722b26295d2aba6b"}
Sep 16 19:38:36.218207 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:36.218171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:36.367934 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:36.367849    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:36.779739 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:36.779701    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:36.904697 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:36.904647    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:39.649201 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:39.649162    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:39.749124 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:39.749090    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:40.582648 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:40.582604    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:40.891119 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:40.891074    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:41.009307 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:41.009233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:42.658237 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:42.658200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerStarted","Data":"afe1e8dccd8d3fe70541c44823022c08d7983860bf8476c44c2c02f29be04454"}
Sep 16 19:38:42.809054 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:42.808960    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:42.889603 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:42.889568    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:44.670419 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:44.670377    2566 generic.go:356] "Generic (PLEG): container finished" podID="786faea0-23df-4f14-9db8-42066b13ed76" containerID="afe1e8dccd8d3fe70541c44823022c08d7983860bf8476c44c2c02f29be04454" exitCode=0
Sep 16 19:38:44.670830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:44.670454    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerDied","Data":"afe1e8dccd8d3fe70541c44823022c08d7983860bf8476c44c2c02f29be04454"}
Sep 16 19:38:46.112073 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:46.112011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:46.387715 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:46.387616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:46.491447 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:46.491394    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:47.668014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:47.667967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:47.765773 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:47.765724    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:51.636623 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:51.636576    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:51.745147 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:51.745094    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:52.730135 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:52.730089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:52.828125 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:52.828079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:57.999940 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:57.999868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:38:58.106949 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:38:58.106908    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:04.792496 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:04.792393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:05.128965 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.128870    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"]
Sep 16 19:39:05.248420 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.248375    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:05.630859 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.630814    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"]
Sep 16 19:39:05.631085 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.630931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.635227 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.635166    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-kserve-self-signed-certs\""
Sep 16 19:39:05.635679 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.635653    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"kube-root-ca.crt\""
Sep 16 19:39:05.635795 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.635724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-j64w8\""
Sep 16 19:39:05.635872 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.635852    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"openshift-service-ca.crt\""
Sep 16 19:39:05.635961 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.635951    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"default-dockercfg-9766v\""
Sep 16 19:39:05.748271 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.748228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ab5640-b133-4176-981c-40eb2a318d53-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.748484 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.748285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g95qv\" (UniqueName: \"kubernetes.io/projected/86ab5640-b133-4176-981c-40eb2a318d53-kube-api-access-g95qv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.849288 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.849228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g95qv\" (UniqueName: \"kubernetes.io/projected/86ab5640-b133-4176-981c-40eb2a318d53-kube-api-access-g95qv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.849816 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.849389    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ab5640-b133-4176-981c-40eb2a318d53-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.852550 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.852525    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ab5640-b133-4176-981c-40eb2a318d53-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.859011 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.858986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g95qv\" (UniqueName: \"kubernetes.io/projected/86ab5640-b133-4176-981c-40eb2a318d53-kube-api-access-g95qv\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:05.945216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:05.945180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:12.259485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:12.259457    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"]
Sep 16 19:39:12.320310 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:39:12.320276    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ab5640_b133_4176_981c_40eb2a318d53.slice/crio-cbbe1cc13e5380e9053840046ffbfce789de530055a5a191e250ea7b926c1c43 WatchSource:0}: Error finding container cbbe1cc13e5380e9053840046ffbfce789de530055a5a191e250ea7b926c1c43: Status 404 returned error can't find the container with id cbbe1cc13e5380e9053840046ffbfce789de530055a5a191e250ea7b926c1c43
Sep 16 19:39:12.856857 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:12.856757    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" event={"ID":"86ab5640-b133-4176-981c-40eb2a318d53","Type":"ContainerStarted","Data":"bc985adda823b59b81c774ff0877d4b50e408b73040b1e0949629eba88eacee5"}
Sep 16 19:39:12.856857 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:12.856822    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" event={"ID":"86ab5640-b133-4176-981c-40eb2a318d53","Type":"ContainerStarted","Data":"cbbe1cc13e5380e9053840046ffbfce789de530055a5a191e250ea7b926c1c43"}
Sep 16 19:39:12.857124 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:12.856930    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:12.858774 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:12.858752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerStarted","Data":"699f2197c148716d1f67ebdc6a679aecfd883837201be5adb9eb3f46b6b618bc"}
Sep 16 19:39:12.875382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:12.875328    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" podStartSLOduration=7.875311037 podStartE2EDuration="7.875311037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:12.874220769 +0000 UTC m=+3336.986017448" watchObservedRunningTime="2026-09-16 19:39:12.875311037 +0000 UTC m=+3336.987107678"
Sep 16 19:39:13.866969 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:13.866850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerStarted","Data":"55527b83b0d253f220376716a55b299962f52fae7dd618e8f096cb9e72928fe5"}
Sep 16 19:39:13.891216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:13.891091    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podStartSLOduration=3.564964723 podStartE2EDuration="39.891070048s" totalImagesPullingTime="34.941545988s" totalInitContainerRuntime="1.384559337s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:38:34 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:39:13.889479982 +0000 UTC m=+3338.001276647" watchObservedRunningTime="2026-09-16 19:39:13.891070048 +0000 UTC m=+3338.002866703"
Sep 16 19:39:14.166590 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:14.166552    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:39:15.166805 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:15.166763    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:39:15.168337 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:15.168299    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:39:16.528837 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:16.528797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:39:25.166780 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:25.166733    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:39:35.166914 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:35.166854    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:39:43.868521 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:43.868484    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:39:45.167565 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:45.167505    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:39:55.167755 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:39:55.167707    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:40:05.166750 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:05.166706    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:40:10.310899 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:10.310838    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:40:15.167655 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:15.167607    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:40:25.166683 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:25.166640    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:40:35.166828 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:35.166735    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:40:45.167487 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:45.167437    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:40:55.167307 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:40:55.167250    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:41:02.614927 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:02.614867    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:05.166825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:05.166778    2566 prober.go:120] "Probe failed" probeType="Startup" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:41:07.499048 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:07.499013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:07.567143 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:07.567104    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"]
Sep 16 19:41:07.685128 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:07.685093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:07.804736 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:07.804641    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:13.868793 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:13.868752    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:15.176573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:15.176537    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:41:15.184548 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:15.184521    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:41:20.560544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.560211    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:20.777495 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.777455    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"]
Sep 16 19:41:20.781558 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.781525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:20.784295 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.784265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-stop-feature-d9eef2a1\"/\"stop-feature-test-epp-sa-dockercfg-kwwc6\""
Sep 16 19:41:20.797235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.797202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"]
Sep 16 19:41:20.824669 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.824569    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:20.902310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.902271    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3d0ec-cbe1-4273-add2-8a93133a9993-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:20.902489 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:20.902328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crlwj\" (UniqueName: \"kubernetes.io/projected/71b3d0ec-cbe1-4273-add2-8a93133a9993-kube-api-access-crlwj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.003243 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.003196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crlwj\" (UniqueName: \"kubernetes.io/projected/71b3d0ec-cbe1-4273-add2-8a93133a9993-kube-api-access-crlwj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.003435 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.003288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3d0ec-cbe1-4273-add2-8a93133a9993-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.005894 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.005853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3d0ec-cbe1-4273-add2-8a93133a9993-tls-certs\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.011705 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.011679    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crlwj\" (UniqueName: \"kubernetes.io/projected/71b3d0ec-cbe1-4273-add2-8a93133a9993-kube-api-access-crlwj\") pod \"stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") " pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.092667 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.092566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.369640 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.369608    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"]
Sep 16 19:41:21.371150 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:41:21.371122    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b3d0ec_cbe1_4273_add2_8a93133a9993.slice/crio-a466e737366aca9783cb86b0ad85c45947cced551f2099f9b7e689e907bb5874 WatchSource:0}: Error finding container a466e737366aca9783cb86b0ad85c45947cced551f2099f9b7e689e907bb5874: Status 404 returned error can't find the container with id a466e737366aca9783cb86b0ad85c45947cced551f2099f9b7e689e907bb5874
Sep 16 19:41:21.611560 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.611519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" event={"ID":"71b3d0ec-cbe1-4273-add2-8a93133a9993","Type":"ContainerStarted","Data":"086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6"}
Sep 16 19:41:21.612017 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.611577    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" event={"ID":"71b3d0ec-cbe1-4273-add2-8a93133a9993","Type":"ContainerStarted","Data":"a466e737366aca9783cb86b0ad85c45947cced551f2099f9b7e689e907bb5874"}
Sep 16 19:41:21.612017 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.611607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:21.632439 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:21.632315    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" podStartSLOduration=1.6322974220000002 podStartE2EDuration="1.632297422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:21.62970872 +0000 UTC m=+3465.741505361" watchObservedRunningTime="2026-09-16 19:41:21.632297422 +0000 UTC m=+3465.744094063"
Sep 16 19:41:22.587126 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:22.587074    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main" containerID="cri-o://bc985adda823b59b81c774ff0877d4b50e408b73040b1e0949629eba88eacee5" gracePeriod=45
Sep 16 19:41:23.869354 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:23.869310    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:33.868682 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:33.868638    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:33.869150 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:33.868779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:41:43.868906 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:43.868847    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:41:47.618808 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:47.618738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:50.782504 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:50.782459    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:50.783202 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:50.783171    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"]
Sep 16 19:41:50.783656 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:50.783511    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="modelcar" containerID="cri-o://55527b83b0d253f220376716a55b299962f52fae7dd618e8f096cb9e72928fe5" gracePeriod=60
Sep 16 19:41:50.857512 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:50.857475    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:41:51.779772 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:51.779740    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db_786faea0-23df-4f14-9db8-42066b13ed76/modelcar/0.log"
Sep 16 19:41:51.780527 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:51.780496    2566 generic.go:356] "Generic (PLEG): container finished" podID="786faea0-23df-4f14-9db8-42066b13ed76" containerID="55527b83b0d253f220376716a55b299962f52fae7dd618e8f096cb9e72928fe5" exitCode=143
Sep 16 19:41:51.780649 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:51.780541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerDied","Data":"55527b83b0d253f220376716a55b299962f52fae7dd618e8f096cb9e72928fe5"}
Sep 16 19:41:52.619409 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.619379    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:41:52.790071 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.790025    2566 generic.go:356] "Generic (PLEG): container finished" podID="86ab5640-b133-4176-981c-40eb2a318d53" containerID="bc985adda823b59b81c774ff0877d4b50e408b73040b1e0949629eba88eacee5" exitCode=0
Sep 16 19:41:52.790235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.790095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" event={"ID":"86ab5640-b133-4176-981c-40eb2a318d53","Type":"ContainerDied","Data":"bc985adda823b59b81c774ff0877d4b50e408b73040b1e0949629eba88eacee5"}
Sep 16 19:41:52.828714 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.828685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:41:52.884338 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.884235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g95qv\" (UniqueName: \"kubernetes.io/projected/86ab5640-b133-4176-981c-40eb2a318d53-kube-api-access-g95qv\") pod \"86ab5640-b133-4176-981c-40eb2a318d53\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") "
Sep 16 19:41:52.884502 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.884399    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ab5640-b133-4176-981c-40eb2a318d53-tls-certs\") pod \"86ab5640-b133-4176-981c-40eb2a318d53\" (UID: \"86ab5640-b133-4176-981c-40eb2a318d53\") "
Sep 16 19:41:52.886467 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.886442    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ab5640-b133-4176-981c-40eb2a318d53-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86ab5640-b133-4176-981c-40eb2a318d53" (UID: "86ab5640-b133-4176-981c-40eb2a318d53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:41:52.886550 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.886525    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ab5640-b133-4176-981c-40eb2a318d53-kube-api-access-g95qv" (OuterVolumeSpecName: "kube-api-access-g95qv") pod "86ab5640-b133-4176-981c-40eb2a318d53" (UID: "86ab5640-b133-4176-981c-40eb2a318d53"). InnerVolumeSpecName "kube-api-access-g95qv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:41:52.985940 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.985899    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ab5640-b133-4176-981c-40eb2a318d53-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:41:52.985940 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:52.985940    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g95qv\" (UniqueName: \"kubernetes.io/projected/86ab5640-b133-4176-981c-40eb2a318d53-kube-api-access-g95qv\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:41:53.796528 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:53.796490    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"
Sep 16 19:41:53.797082 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:53.796532    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn" event={"ID":"86ab5640-b133-4176-981c-40eb2a318d53","Type":"ContainerDied","Data":"cbbe1cc13e5380e9053840046ffbfce789de530055a5a191e250ea7b926c1c43"}
Sep 16 19:41:53.797082 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:53.796607    2566 scope.go:117] "RemoveContainer" containerID="bc985adda823b59b81c774ff0877d4b50e408b73040b1e0949629eba88eacee5"
Sep 16 19:41:53.822893 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:53.822847    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"]
Sep 16 19:41:53.824952 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:53.824929    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-lsntn"]
Sep 16 19:41:54.403304 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:54.403264    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ab5640-b133-4176-981c-40eb2a318d53" path="/var/lib/kubelet/pods/86ab5640-b133-4176-981c-40eb2a318d53/volumes"
Sep 16 19:41:57.199701 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:41:57.199650    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:05.807837 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:05.807726    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" containerID="cri-o://699f2197c148716d1f67ebdc6a679aecfd883837201be5adb9eb3f46b6b618bc" gracePeriod=45
Sep 16 19:42:06.178252 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:06.178199    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Sep 16 19:42:06.870565 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:06.870536    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db_786faea0-23df-4f14-9db8-42066b13ed76/modelcar/0.log"
Sep 16 19:42:06.871383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:06.871352    2566 generic.go:356] "Generic (PLEG): container finished" podID="786faea0-23df-4f14-9db8-42066b13ed76" containerID="699f2197c148716d1f67ebdc6a679aecfd883837201be5adb9eb3f46b6b618bc" exitCode=0
Sep 16 19:42:06.871515 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:06.871403    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerDied","Data":"699f2197c148716d1f67ebdc6a679aecfd883837201be5adb9eb3f46b6b618bc"}
Sep 16 19:42:07.071689 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.071626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db_786faea0-23df-4f14-9db8-42066b13ed76/modelcar/0.log"
Sep 16 19:42:07.074545 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.074524    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:42:07.221356 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221313    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786faea0-23df-4f14-9db8-42066b13ed76-tls-certs\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221375    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-dshm\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221573 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl5d9\" (UniqueName: \"kubernetes.io/projected/786faea0-23df-4f14-9db8-42066b13ed76-kube-api-access-zl5d9\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221694 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221575    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-model-cache\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221694 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-kserve-provision-location\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221694 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-tmp-dir\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221694 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221663    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-home\") pod \"786faea0-23df-4f14-9db8-42066b13ed76\" (UID: \"786faea0-23df-4f14-9db8-42066b13ed76\") "
Sep 16 19:42:07.221959 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-model-cache" (OuterVolumeSpecName: "model-cache") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:07.222029 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.221981    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:07.222502 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.222476    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-home" (OuterVolumeSpecName: "home") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:07.223861 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.223842    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-dshm" (OuterVolumeSpecName: "dshm") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:07.223971 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.223947    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/786faea0-23df-4f14-9db8-42066b13ed76-kube-api-access-zl5d9" (OuterVolumeSpecName: "kube-api-access-zl5d9") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "kube-api-access-zl5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:42:07.224039 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.224017    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/786faea0-23df-4f14-9db8-42066b13ed76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:42:07.233145 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.233117    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "786faea0-23df-4f14-9db8-42066b13ed76" (UID: "786faea0-23df-4f14-9db8-42066b13ed76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323061    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl5d9\" (UniqueName: \"kubernetes.io/projected/786faea0-23df-4f14-9db8-42066b13ed76-kube-api-access-zl5d9\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323104    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-model-cache\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323118    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-kserve-provision-location\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323132    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-tmp-dir\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323145    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-home\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323156    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/786faea0-23df-4f14-9db8-42066b13ed76-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.323168 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.323168    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/786faea0-23df-4f14-9db8-42066b13ed76-dshm\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:42:07.877654 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.877623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-test-llm-inference-service-7bc4bd10_llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db_786faea0-23df-4f14-9db8-42066b13ed76/modelcar/0.log"
Sep 16 19:42:07.878492 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.878470    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"
Sep 16 19:42:07.878578 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.878469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db" event={"ID":"786faea0-23df-4f14-9db8-42066b13ed76","Type":"ContainerDied","Data":"05ab445ef193b29e0ac1a0849d8b67c88e820c8004570f66722b26295d2aba6b"}
Sep 16 19:42:07.878646 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.878632    2566 scope.go:117] "RemoveContainer" containerID="55527b83b0d253f220376716a55b299962f52fae7dd618e8f096cb9e72928fe5"
Sep 16 19:42:07.888717 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.888697    2566 scope.go:117] "RemoveContainer" containerID="699f2197c148716d1f67ebdc6a679aecfd883837201be5adb9eb3f46b6b618bc"
Sep 16 19:42:07.896850 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.896795    2566 scope.go:117] "RemoveContainer" containerID="afe1e8dccd8d3fe70541c44823022c08d7983860bf8476c44c2c02f29be04454"
Sep 16 19:42:07.911602 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.911542    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"]
Sep 16 19:42:07.921311 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:07.921275    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-7bc4bd10/llmisvc-model-fb-opt-125m-oci-r-0fba19de-kserve-549cb45f5dk52db"]
Sep 16 19:42:08.402996 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:08.402959    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="786faea0-23df-4f14-9db8-42066b13ed76" path="/var/lib/kubelet/pods/786faea0-23df-4f14-9db8-42066b13ed76/volumes"
Sep 16 19:42:38.880659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:38.880617    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:38.988164 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:38.988115    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:45.476735 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:45.476693    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:46.908843 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:46.908804    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:47.181194 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:47.181096    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:51.537989 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:51.537947    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:51.690570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:51.690531    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:42:56.017081 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:42:56.017039    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.691327 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:12.691279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:12.703390 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:12.703356    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"]
Sep 16 19:43:12.785138 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:12.785096    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:43:22.619485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:22.619435    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:27.722505 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:27.722443    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main" containerID="cri-o://086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6" gracePeriod=45
Sep 16 19:43:32.619181 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:32.619085    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:42.619411 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:42.619360    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:42.619827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:42.619486    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:43:52.619014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:52.618971    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:43:57.976364 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:57.976334    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:43:58.077308 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.077195    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crlwj\" (UniqueName: \"kubernetes.io/projected/71b3d0ec-cbe1-4273-add2-8a93133a9993-kube-api-access-crlwj\") pod \"71b3d0ec-cbe1-4273-add2-8a93133a9993\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") "
Sep 16 19:43:58.077506 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.077329    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3d0ec-cbe1-4273-add2-8a93133a9993-tls-certs\") pod \"71b3d0ec-cbe1-4273-add2-8a93133a9993\" (UID: \"71b3d0ec-cbe1-4273-add2-8a93133a9993\") "
Sep 16 19:43:58.079406 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.079368    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71b3d0ec-cbe1-4273-add2-8a93133a9993-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "71b3d0ec-cbe1-4273-add2-8a93133a9993" (UID: "71b3d0ec-cbe1-4273-add2-8a93133a9993"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:43:58.079519 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.079424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b3d0ec-cbe1-4273-add2-8a93133a9993-kube-api-access-crlwj" (OuterVolumeSpecName: "kube-api-access-crlwj") pod "71b3d0ec-cbe1-4273-add2-8a93133a9993" (UID: "71b3d0ec-cbe1-4273-add2-8a93133a9993"). InnerVolumeSpecName "kube-api-access-crlwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:43:58.178832 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.178790    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71b3d0ec-cbe1-4273-add2-8a93133a9993-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:43:58.178832 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.178826    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crlwj\" (UniqueName: \"kubernetes.io/projected/71b3d0ec-cbe1-4273-add2-8a93133a9993-kube-api-access-crlwj\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:43:58.461255 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.461219    2566 generic.go:356] "Generic (PLEG): container finished" podID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerID="086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6" exitCode=0
Sep 16 19:43:58.461436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.461267    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"
Sep 16 19:43:58.461436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.461301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" event={"ID":"71b3d0ec-cbe1-4273-add2-8a93133a9993","Type":"ContainerDied","Data":"086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6"}
Sep 16 19:43:58.461436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.461356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t" event={"ID":"71b3d0ec-cbe1-4273-add2-8a93133a9993","Type":"ContainerDied","Data":"a466e737366aca9783cb86b0ad85c45947cced551f2099f9b7e689e907bb5874"}
Sep 16 19:43:58.461436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.461389    2566 scope.go:117] "RemoveContainer" containerID="086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6"
Sep 16 19:43:58.470259 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.470242    2566 scope.go:117] "RemoveContainer" containerID="086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6"
Sep 16 19:43:58.470529 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:43:58.470510    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6\": container with ID starting with 086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6 not found: ID does not exist" containerID="086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6"
Sep 16 19:43:58.470603 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.470537    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6"} err="failed to get container status \"086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6\": rpc error: code = NotFound desc = could not find container \"086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6\": container with ID starting with 086bd837c3b6e182b2c8a9a22820af66a84664afcebac0c18278cc5aa8834ab6 not found: ID does not exist"
Sep 16 19:43:58.477693 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.477662    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"]
Sep 16 19:43:58.480121 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:43:58.480097    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-stop-feature-d9eef2a1/stop-feature-test-kserve-router-scheduler-7cc9bbb8f8-hd67t"]
Sep 16 19:44:00.402638 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:00.402604    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" path="/var/lib/kubelet/pods/71b3d0ec-cbe1-4273-add2-8a93133a9993/volumes"
Sep 16 19:44:02.604775 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:02.604736    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:02.704399 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:02.704357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:02.808188 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:02.808154    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:08.822551 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:08.822510    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:14.863487 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:14.863440    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:15.072210 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:15.072166    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:26.256986 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:26.256714    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:52.102813 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:52.102773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:44:52.170759 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:44:52.170718    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:04.401179 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:04.401089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:04.575478 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:04.575439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:06.110319 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:06.110280    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:06.168682 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:06.168646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:42.295073 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:42.295031    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:42.393589 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:42.393547    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:42.550726 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:42.550586    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:42.649355 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:42.649311    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:46.408277 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:46.408240    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:48.076562 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:48.076521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:49.708284 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:49.708243    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:55.374686 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:55.374648    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:55.475383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:55.475340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:58.176546 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.176496    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:58.497458 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497365    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"]
Sep 16 19:45:58.497825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497828    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497844    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="modelcar-init"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497852    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="modelcar-init"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497871    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497895    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497908    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497917    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497926    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="modelcar"
Sep 16 19:45:58.497933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.497935    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="modelcar"
Sep 16 19:45:58.498387 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.498025    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="modelcar"
Sep 16 19:45:58.498387 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.498040    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b3d0ec-cbe1-4273-add2-8a93133a9993" containerName="main"
Sep 16 19:45:58.498387 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.498049    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="786faea0-23df-4f14-9db8-42066b13ed76" containerName="main"
Sep 16 19:45:58.498387 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.498058    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ab5640-b133-4176-981c-40eb2a318d53" containerName="main"
Sep 16 19:45:58.501075 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.501052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.504926 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.504902    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"kube-root-ca.crt\""
Sep 16 19:45:58.505849 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.505830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-epp-sa-dockercfg-bgc46\""
Sep 16 19:45:58.506436 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.506416    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"openshift-service-ca.crt\""
Sep 16 19:45:58.506559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.506416    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"default-dockercfg-fr6f2\""
Sep 16 19:45:58.506559 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.506419    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-028f7809\"/\"router-with-refs-test-kserve-self-signed-certs\""
Sep 16 19:45:58.514608 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.514581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"]
Sep 16 19:45:58.614803 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.614755    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f593287-4533-4816-8ce1-279ddd848a76-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.615038 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.614824    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7kws\" (UniqueName: \"kubernetes.io/projected/7f593287-4533-4816-8ce1-279ddd848a76-kube-api-access-z7kws\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.715680 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.715640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7kws\" (UniqueName: \"kubernetes.io/projected/7f593287-4533-4816-8ce1-279ddd848a76-kube-api-access-z7kws\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.715855 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.715713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f593287-4533-4816-8ce1-279ddd848a76-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.718231 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.718200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f593287-4533-4816-8ce1-279ddd848a76-tls-certs\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.725709 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.725682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7kws\" (UniqueName: \"kubernetes.io/projected/7f593287-4533-4816-8ce1-279ddd848a76-kube-api-access-z7kws\") pod \"router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") " pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.810873 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.810780    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:45:58.967868 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:58.967818    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:59.092514 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:59.092420    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:45:59.101435 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:45:59.101406    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"]
Sep 16 19:45:59.103206 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:45:59.103180    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f593287_4533_4816_8ce1_279ddd848a76.slice/crio-3d40ce60bd8e6ce0ed7bb2736e64f3153653de8c9a2d2b7af66f200c6a21d5bb WatchSource:0}: Error finding container 3d40ce60bd8e6ce0ed7bb2736e64f3153653de8c9a2d2b7af66f200c6a21d5bb: Status 404 returned error can't find the container with id 3d40ce60bd8e6ce0ed7bb2736e64f3153653de8c9a2d2b7af66f200c6a21d5bb
Sep 16 19:46:00.097825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:00.097789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" event={"ID":"7f593287-4533-4816-8ce1-279ddd848a76","Type":"ContainerStarted","Data":"53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e"}
Sep 16 19:46:00.098251 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:00.097843    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" event={"ID":"7f593287-4533-4816-8ce1-279ddd848a76","Type":"ContainerStarted","Data":"3d40ce60bd8e6ce0ed7bb2736e64f3153653de8c9a2d2b7af66f200c6a21d5bb"}
Sep 16 19:46:00.098251 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:00.097864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:46:00.123015 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:00.122949    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" podStartSLOduration=2.122913047 podStartE2EDuration="2.122913047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:46:00.120579561 +0000 UTC m=+3744.232376202" watchObservedRunningTime="2026-09-16 19:46:00.122913047 +0000 UTC m=+3744.234709685"
Sep 16 19:46:31.104458 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:31.104376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:46:39.143111 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:39.143066    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:59.039195 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:59.039156    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:59.143035 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:59.142995    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:46:59.242900 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:46:59.242855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:03.369230 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:47:03.369177    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:05.574402 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:47:05.574371    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:47:05.653433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:47:05.653398    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:03.261355 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:03.261250    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:03.271625 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:03.271589    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"]
Sep 16 19:48:03.338579 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:03.338530    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:11.104220 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:11.104173    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:48:18.291451 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:18.291407    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main" containerID="cri-o://53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e" gracePeriod=45
Sep 16 19:48:21.103978 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:21.103937    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:48:31.104336 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:31.104291    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:48:31.104758 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:31.104413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:48:41.104835 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:41.104792    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:48:48.542398 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.542368    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:48:48.660105 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.660070    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f593287-4533-4816-8ce1-279ddd848a76-tls-certs\") pod \"7f593287-4533-4816-8ce1-279ddd848a76\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") "
Sep 16 19:48:48.660275 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.660156    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7kws\" (UniqueName: \"kubernetes.io/projected/7f593287-4533-4816-8ce1-279ddd848a76-kube-api-access-z7kws\") pod \"7f593287-4533-4816-8ce1-279ddd848a76\" (UID: \"7f593287-4533-4816-8ce1-279ddd848a76\") "
Sep 16 19:48:48.662376 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.662350    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f593287-4533-4816-8ce1-279ddd848a76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7f593287-4533-4816-8ce1-279ddd848a76" (UID: "7f593287-4533-4816-8ce1-279ddd848a76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:48:48.662457 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.662395    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f593287-4533-4816-8ce1-279ddd848a76-kube-api-access-z7kws" (OuterVolumeSpecName: "kube-api-access-z7kws") pod "7f593287-4533-4816-8ce1-279ddd848a76" (UID: "7f593287-4533-4816-8ce1-279ddd848a76"). InnerVolumeSpecName "kube-api-access-z7kws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:48:48.761718 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.761672    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f593287-4533-4816-8ce1-279ddd848a76-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:48:48.761718 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:48.761709    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7kws\" (UniqueName: \"kubernetes.io/projected/7f593287-4533-4816-8ce1-279ddd848a76-kube-api-access-z7kws\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:48:49.000830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.000705    2566 generic.go:356] "Generic (PLEG): container finished" podID="7f593287-4533-4816-8ce1-279ddd848a76" containerID="53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e" exitCode=0
Sep 16 19:48:49.000830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.000762    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"
Sep 16 19:48:49.000830 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.000797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" event={"ID":"7f593287-4533-4816-8ce1-279ddd848a76","Type":"ContainerDied","Data":"53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e"}
Sep 16 19:48:49.001191 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.000866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz" event={"ID":"7f593287-4533-4816-8ce1-279ddd848a76","Type":"ContainerDied","Data":"3d40ce60bd8e6ce0ed7bb2736e64f3153653de8c9a2d2b7af66f200c6a21d5bb"}
Sep 16 19:48:49.001191 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.000926    2566 scope.go:117] "RemoveContainer" containerID="53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e"
Sep 16 19:48:49.011216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.011195    2566 scope.go:117] "RemoveContainer" containerID="53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e"
Sep 16 19:48:49.011493 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:48:49.011474    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e\": container with ID starting with 53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e not found: ID does not exist" containerID="53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e"
Sep 16 19:48:49.011560 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.011516    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e"} err="failed to get container status \"53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e\": rpc error: code = NotFound desc = could not find container \"53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e\": container with ID starting with 53e4fc409907cac673c5236d943fe28639e04efa3147cd9379e7a14d727def8e not found: ID does not exist"
Sep 16 19:48:49.025676 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.025646    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"]
Sep 16 19:48:49.029141 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:49.029119    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-028f7809/router-with-refs-test-kserve-router-scheduler-5776bf489f-bx5mz"]
Sep 16 19:48:50.402540 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:50.402505    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f593287-4533-4816-8ce1-279ddd848a76" path="/var/lib/kubelet/pods/7f593287-4533-4816-8ce1-279ddd848a76/volumes"
Sep 16 19:48:50.575061 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:50.575016    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:53.262150 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:53.262109    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:53.408005 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:53.407968    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:48:58.310430 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:48:58.310386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:10.769405 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:10.769365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:10.874593 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:10.874553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:10.956124 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:10.956091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:11.057570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.057459    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:11.158789 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.158734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:11.257474 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.257438    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:11.387101 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.387069    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"]
Sep 16 19:49:11.387455 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.387442    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main"
Sep 16 19:49:11.387516 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.387456    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main"
Sep 16 19:49:11.387554 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.387521    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f593287-4533-4816-8ce1-279ddd848a76" containerName="main"
Sep 16 19:49:11.390452 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.390430    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.393194 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.393174    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"openshift-service-ca.crt\""
Sep 16 19:49:11.394305 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.394282    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"kube-root-ca.crt\""
Sep 16 19:49:11.394537 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.394510    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-epp-sa-dockercfg-d2kr8\""
Sep 16 19:49:11.394659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.394610    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"default-dockercfg-hsx46\""
Sep 16 19:49:11.394659 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.394634    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-62876e81\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Sep 16 19:49:11.402933 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.402908    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"]
Sep 16 19:49:11.497420 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.497385    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:11.565685 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.565639    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv6bw\" (UniqueName: \"kubernetes.io/projected/944decbf-c193-4bf7-b022-8ad6243b7dc0-kube-api-access-mv6bw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.565866 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.565706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/944decbf-c193-4bf7-b022-8ad6243b7dc0-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.666608 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.666516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/944decbf-c193-4bf7-b022-8ad6243b7dc0-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.666608 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.666590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv6bw\" (UniqueName: \"kubernetes.io/projected/944decbf-c193-4bf7-b022-8ad6243b7dc0-kube-api-access-mv6bw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.669421 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.669389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/944decbf-c193-4bf7-b022-8ad6243b7dc0-tls-certs\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.676719 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.676694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv6bw\" (UniqueName: \"kubernetes.io/projected/944decbf-c193-4bf7-b022-8ad6243b7dc0-kube-api-access-mv6bw\") pod \"llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") " pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.701531 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.701498    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:11.980214 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:11.980180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"]
Sep 16 19:49:11.981547 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:49:11.981518    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod944decbf_c193_4bf7_b022_8ad6243b7dc0.slice/crio-afb624e550428cbd6307ba353d49b53409b108aa4ce2ea922c1d8345aad5cf8c WatchSource:0}: Error finding container afb624e550428cbd6307ba353d49b53409b108aa4ce2ea922c1d8345aad5cf8c: Status 404 returned error can't find the container with id afb624e550428cbd6307ba353d49b53409b108aa4ce2ea922c1d8345aad5cf8c
Sep 16 19:49:12.127353 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:12.127317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" event={"ID":"944decbf-c193-4bf7-b022-8ad6243b7dc0","Type":"ContainerStarted","Data":"c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba"}
Sep 16 19:49:12.127549 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:12.127370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" event={"ID":"944decbf-c193-4bf7-b022-8ad6243b7dc0","Type":"ContainerStarted","Data":"afb624e550428cbd6307ba353d49b53409b108aa4ce2ea922c1d8345aad5cf8c"}
Sep 16 19:49:12.127549 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:12.127409    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:49:12.147383 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:12.147325    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" podStartSLOduration=1.147305853 podStartE2EDuration="1.147305853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:49:12.145366805 +0000 UTC m=+3936.257163449" watchObservedRunningTime="2026-09-16 19:49:12.147305853 +0000 UTC m=+3936.259102493"
Sep 16 19:49:13.527114 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:13.527073    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:13.628626 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:13.628583    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:14.183496 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:14.183458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:17.006671 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:17.006633    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:17.112867 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:17.112828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:19.420022 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:19.419986    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:19.701125 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:19.701025    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:21.812458 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:21.812409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:28.626077 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:28.625971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:34.972133 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:34.972077    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:36.301104 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:36.301066    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:41.950967 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:41.950930    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:42.053485 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:42.053446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:49:43.134451 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:49:43.134414    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:50:07.805825 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:50:07.805781    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:35.762893 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:50:35.762853    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:50:35.867548 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:50:35.867508    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:35.182326 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:35.182284    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:35.204095 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:35.204060    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"]
Sep 16 19:51:35.280406 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:35.280365    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:40.413529 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:40.413488    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:51:43.134507 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:43.134466    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:51:50.223790 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:50.223746    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main" containerID="cri-o://c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba" gracePeriod=45
Sep 16 19:51:53.133731 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:51:53.133686    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:52:00.545503 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:00.545461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:00.650270 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:00.650233    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:00.746773 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:00.746728    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:00.828215 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:00.828122    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:00.914653 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:00.914616    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:03.110444 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:03.110406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:03.134085 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:03.134047    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:52:03.134257 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:03.134145    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:52:03.208253 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:03.208220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:03.942449 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:03.942407    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:05.780698 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:05.780649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:05.880307 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:05.880262    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:06.947526 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:06.947483    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:07.047701 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:07.047656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:07.366362 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:07.366245    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:13.103408 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:13.103363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:13.134411 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:13.134371    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:52:13.255475 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:13.255442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:20.474049 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.474013    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:52:20.514294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.514203    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mv6bw\" (UniqueName: \"kubernetes.io/projected/944decbf-c193-4bf7-b022-8ad6243b7dc0-kube-api-access-mv6bw\") pod \"944decbf-c193-4bf7-b022-8ad6243b7dc0\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") "
Sep 16 19:52:20.514294 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.514268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/944decbf-c193-4bf7-b022-8ad6243b7dc0-tls-certs\") pod \"944decbf-c193-4bf7-b022-8ad6243b7dc0\" (UID: \"944decbf-c193-4bf7-b022-8ad6243b7dc0\") "
Sep 16 19:52:20.516512 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.516479    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944decbf-c193-4bf7-b022-8ad6243b7dc0-kube-api-access-mv6bw" (OuterVolumeSpecName: "kube-api-access-mv6bw") pod "944decbf-c193-4bf7-b022-8ad6243b7dc0" (UID: "944decbf-c193-4bf7-b022-8ad6243b7dc0"). InnerVolumeSpecName "kube-api-access-mv6bw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:52:20.516512 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.516484    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/944decbf-c193-4bf7-b022-8ad6243b7dc0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "944decbf-c193-4bf7-b022-8ad6243b7dc0" (UID: "944decbf-c193-4bf7-b022-8ad6243b7dc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:52:20.614826 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.614784    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv6bw\" (UniqueName: \"kubernetes.io/projected/944decbf-c193-4bf7-b022-8ad6243b7dc0-kube-api-access-mv6bw\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:52:20.614826 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:20.614821    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/944decbf-c193-4bf7-b022-8ad6243b7dc0-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:52:21.117334 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.117296    2566 generic.go:356] "Generic (PLEG): container finished" podID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerID="c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba" exitCode=0
Sep 16 19:52:21.117524 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.117349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" event={"ID":"944decbf-c193-4bf7-b022-8ad6243b7dc0","Type":"ContainerDied","Data":"c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba"}
Sep 16 19:52:21.117524 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.117354    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"
Sep 16 19:52:21.117524 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.117417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj" event={"ID":"944decbf-c193-4bf7-b022-8ad6243b7dc0","Type":"ContainerDied","Data":"afb624e550428cbd6307ba353d49b53409b108aa4ce2ea922c1d8345aad5cf8c"}
Sep 16 19:52:21.117524 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.117441    2566 scope.go:117] "RemoveContainer" containerID="c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba"
Sep 16 19:52:21.126746 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.126724    2566 scope.go:117] "RemoveContainer" containerID="c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba"
Sep 16 19:52:21.127052 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:52:21.127028    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba\": container with ID starting with c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba not found: ID does not exist" containerID="c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba"
Sep 16 19:52:21.127154 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.127059    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba"} err="failed to get container status \"c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba\": rpc error: code = NotFound desc = could not find container \"c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba\": container with ID starting with c402d0d740efb0688a16b78c3fe6dab786dedf78bc06c338f7aafd43651792ba not found: ID does not exist"
Sep 16 19:52:21.140850 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.140818    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"]
Sep 16 19:52:21.145046 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:21.145007    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-62876e81/llmisvc44d181485fad85e662eb092f3749502f-kserve-router-schebdvdj"]
Sep 16 19:52:22.402992 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:22.402946    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" path="/var/lib/kubelet/pods/944decbf-c193-4bf7-b022-8ad6243b7dc0/volumes"
Sep 16 19:52:25.330734 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:25.330694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:25.431678 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:25.431639    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:29.292612 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:29.292527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:32.474270 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:32.474226    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:38.159543 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:38.159501    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:38.257396 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:38.257359    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:44.061905 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.061850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:44.308014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.307975    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"]
Sep 16 19:52:44.308356 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.308342    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main"
Sep 16 19:52:44.308401 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.308359    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main"
Sep 16 19:52:44.308437 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.308423    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="944decbf-c193-4bf7-b022-8ad6243b7dc0" containerName="main"
Sep 16 19:52:44.312874 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.312801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.317514 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.317480    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"kube-root-ca.crt\""
Sep 16 19:52:44.317672 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.317532    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Sep 16 19:52:44.317672 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.317560    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"default-dockercfg-cm5jg\""
Sep 16 19:52:44.317672 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.317569    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"openshift-service-ca.crt\""
Sep 16 19:52:44.317836 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.317713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-d73d44f4\"/\"custom-route-timeout-pd-test-epp-sa-dockercfg-c5xjr\""
Sep 16 19:52:44.325984 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.325955    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"]
Sep 16 19:52:44.338650 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.338615    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:52:44.430487 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.430451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.430658 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.430510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mg5j\" (UniqueName: \"kubernetes.io/projected/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-kube-api-access-7mg5j\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.531515 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.531470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.531515 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.531524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mg5j\" (UniqueName: \"kubernetes.io/projected/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-kube-api-access-7mg5j\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.534112 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.534091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.541484 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.541450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mg5j\" (UniqueName: \"kubernetes.io/projected/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-kube-api-access-7mg5j\") pod \"custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") " pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.623791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.623689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:44.897066 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:44.897035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"]
Sep 16 19:52:44.898957 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:52:44.898927    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb882ec_cd8b_4e48_b06f_f259d5b3f199.slice/crio-27a0450dcbb729df8cc65dcafc52e6aa6cb5ffb199b135e73b9cf8b45ffe9ef6 WatchSource:0}: Error finding container 27a0450dcbb729df8cc65dcafc52e6aa6cb5ffb199b135e73b9cf8b45ffe9ef6: Status 404 returned error can't find the container with id 27a0450dcbb729df8cc65dcafc52e6aa6cb5ffb199b135e73b9cf8b45ffe9ef6
Sep 16 19:52:45.248397 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:45.248293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" event={"ID":"cbb882ec-cd8b-4e48-b06f-f259d5b3f199","Type":"ContainerStarted","Data":"0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c"}
Sep 16 19:52:45.248397 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:45.248354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" event={"ID":"cbb882ec-cd8b-4e48-b06f-f259d5b3f199","Type":"ContainerStarted","Data":"27a0450dcbb729df8cc65dcafc52e6aa6cb5ffb199b135e73b9cf8b45ffe9ef6"}
Sep 16 19:52:45.248397 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:45.248382    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:52:45.268380 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:52:45.268316    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podStartSLOduration=1.2682973849999999 podStartE2EDuration="1.268297385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:52:45.267217262 +0000 UTC m=+4149.379013939" watchObservedRunningTime="2026-09-16 19:52:45.268297385 +0000 UTC m=+4149.380094027"
Sep 16 19:53:02.656245 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:53:02.656194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:53:16.255779 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:53:16.255740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:54:43.570794 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:54:43.570734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:59.328980 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:54:59.328942    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:54:59.390971 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:54:59.390931    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:03.658806 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:03.658763    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:03.751491 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:03.751449    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:09.216544 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:09.216486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:09.316385 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:09.316344    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:09.540359 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:09.540254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:12.394470 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:12.394439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:12.455780 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:12.455744    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:13.339957 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.339914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:13.636139 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.636105    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"]
Sep 16 19:55:13.639696 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.639669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.643509 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.643477    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"openshift-service-ca.crt\""
Sep 16 19:55:13.643681 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.643580    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"default-dockercfg-qtj6f\""
Sep 16 19:55:13.643681 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.643643    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"kube-root-ca.crt\""
Sep 16 19:55:13.643827 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.643706    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-kserve-self-signed-certs\""
Sep 16 19:55:13.643928 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.643913    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-tls-resources-1d9e6e79\"/\"tls-verification-test-epp-sa-dockercfg-9shp5\""
Sep 16 19:55:13.649648 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.649621    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"]
Sep 16 19:55:13.665396 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.665364    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:13.728913 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.728868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q84c2\" (UniqueName: \"kubernetes.io/projected/8a7a64ce-dcd9-430c-9b7b-f2796660979f-kube-api-access-q84c2\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.729112 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.728964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7a64ce-dcd9-430c-9b7b-f2796660979f-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.829926 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.829857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7a64ce-dcd9-430c-9b7b-f2796660979f-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.830142 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.830012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q84c2\" (UniqueName: \"kubernetes.io/projected/8a7a64ce-dcd9-430c-9b7b-f2796660979f-kube-api-access-q84c2\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.832539 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.832500    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7a64ce-dcd9-430c-9b7b-f2796660979f-tls-certs\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.839683 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.839650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q84c2\" (UniqueName: \"kubernetes.io/projected/8a7a64ce-dcd9-430c-9b7b-f2796660979f-kube-api-access-q84c2\") pod \"tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") " pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:13.952242 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:13.952138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:14.228530 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:14.228501    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"]
Sep 16 19:55:14.230410 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:55:14.230378    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7a64ce_dcd9_430c_9b7b_f2796660979f.slice/crio-6a4b13111cc8d88e38a20dc01bc19bc9a09940237270d455d5c138ec1a81063a WatchSource:0}: Error finding container 6a4b13111cc8d88e38a20dc01bc19bc9a09940237270d455d5c138ec1a81063a: Status 404 returned error can't find the container with id 6a4b13111cc8d88e38a20dc01bc19bc9a09940237270d455d5c138ec1a81063a
Sep 16 19:55:15.032621 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:15.032576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" event={"ID":"8a7a64ce-dcd9-430c-9b7b-f2796660979f","Type":"ContainerStarted","Data":"c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e"}
Sep 16 19:55:15.033170 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:15.032639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" event={"ID":"8a7a64ce-dcd9-430c-9b7b-f2796660979f","Type":"ContainerStarted","Data":"6a4b13111cc8d88e38a20dc01bc19bc9a09940237270d455d5c138ec1a81063a"}
Sep 16 19:55:15.033170 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:15.032668    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:15.093577 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:15.093497    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" podStartSLOduration=2.093476701 podStartE2EDuration="2.093476701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:55:15.091422254 +0000 UTC m=+4299.203218898" watchObservedRunningTime="2026-09-16 19:55:15.093476701 +0000 UTC m=+4299.205273344"
Sep 16 19:55:16.189530 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:16.189489    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:16.289524 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:16.289461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:19.775714 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:19.775678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:31.999000 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:31.998899    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"]
Sep 16 19:55:32.041252 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:32.041207    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:32.107931 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:32.107896    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:55:36.255950 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:36.255904    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:46.040706 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:46.040673    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:55:46.255468 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:46.255424    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:47.019333 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:47.019291    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main" containerID="cri-o://0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c" gracePeriod=45
Sep 16 19:55:56.255175 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:56.255133    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:55:56.255604 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:55:56.255277    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:56:06.255517 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:06.255467    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:16.255265 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:16.255225    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:56:17.269410 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.269379    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:56:17.371586 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.371490    2566 generic.go:356] "Generic (PLEG): container finished" podID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerID="0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c" exitCode=0
Sep 16 19:56:17.371586 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.371532    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"
Sep 16 19:56:17.371794 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.371580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" event={"ID":"cbb882ec-cd8b-4e48-b06f-f259d5b3f199","Type":"ContainerDied","Data":"0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c"}
Sep 16 19:56:17.371794 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.371636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57" event={"ID":"cbb882ec-cd8b-4e48-b06f-f259d5b3f199","Type":"ContainerDied","Data":"27a0450dcbb729df8cc65dcafc52e6aa6cb5ffb199b135e73b9cf8b45ffe9ef6"}
Sep 16 19:56:17.371794 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.371664    2566 scope.go:117] "RemoveContainer" containerID="0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c"
Sep 16 19:56:17.380398 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.380372    2566 scope.go:117] "RemoveContainer" containerID="0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c"
Sep 16 19:56:17.380664 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:56:17.380644    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c\": container with ID starting with 0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c not found: ID does not exist" containerID="0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c"
Sep 16 19:56:17.380712 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.380670    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c"} err="failed to get container status \"0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c\": rpc error: code = NotFound desc = could not find container \"0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c\": container with ID starting with 0f84c4a079f1bf98d14c0bbe8b9b4c4d38a66e5209dcb96e0e9806ee66f4bd2c not found: ID does not exist"
Sep 16 19:56:17.386056 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.386036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-tls-certs\") pod \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") "
Sep 16 19:56:17.386138 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.386093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mg5j\" (UniqueName: \"kubernetes.io/projected/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-kube-api-access-7mg5j\") pod \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\" (UID: \"cbb882ec-cd8b-4e48-b06f-f259d5b3f199\") "
Sep 16 19:56:17.388310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.388284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cbb882ec-cd8b-4e48-b06f-f259d5b3f199" (UID: "cbb882ec-cd8b-4e48-b06f-f259d5b3f199"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:56:17.388423 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.388294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-kube-api-access-7mg5j" (OuterVolumeSpecName: "kube-api-access-7mg5j") pod "cbb882ec-cd8b-4e48-b06f-f259d5b3f199" (UID: "cbb882ec-cd8b-4e48-b06f-f259d5b3f199"). InnerVolumeSpecName "kube-api-access-7mg5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:56:17.487675 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.487631    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mg5j\" (UniqueName: \"kubernetes.io/projected/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-kube-api-access-7mg5j\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:56:17.487675 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.487668    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbb882ec-cd8b-4e48-b06f-f259d5b3f199-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:56:17.695441 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.695404    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"]
Sep 16 19:56:17.699634 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:17.699602    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-d73d44f4/custom-route-timeout-pd-test-kserve-router-scheduler-84c488hp57"]
Sep 16 19:56:18.402522 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:18.402483    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" path="/var/lib/kubelet/pods/cbb882ec-cd8b-4e48-b06f-f259d5b3f199/volumes"
Sep 16 19:56:22.256622 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:22.256581    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:22.414927 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:22.414892    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:22.627715 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:22.627664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:25.900071 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:25.900035    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:26.105691 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:26.105659    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:29.529624 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:29.529585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:35.494466 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:35.494430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:35.565727 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:35.565694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:42.800328 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:42.800290    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:43.062728 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.062629    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"]
Sep 16 19:56:43.063178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.063154    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main"
Sep 16 19:56:43.063178 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.063178    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main"
Sep 16 19:56:43.063369 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.063270    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbb882ec-cd8b-4e48-b06f-f259d5b3f199" containerName="main"
Sep 16 19:56:43.067367 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.067345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.071094 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.071061    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"openshift-service-ca.crt\""
Sep 16 19:56:43.071216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.071080    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Sep 16 19:56:43.071216 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.071140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"default-dockercfg-4m4fc\""
Sep 16 19:56:43.071385 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.071367    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"kube-root-ca.crt\""
Sep 16 19:56:43.071453 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.071430    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-llm-inference-service-b5dd93e6\"/\"router-with-refs-pd-test-epp-sa-dockercfg-b5j7c\""
Sep 16 19:56:43.079606 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.079571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"]
Sep 16 19:56:43.186225 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.186181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3514fae-e2c5-4688-8791-adb34c8175c4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.186391 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.186240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2tth\" (UniqueName: \"kubernetes.io/projected/a3514fae-e2c5-4688-8791-adb34c8175c4-kube-api-access-g2tth\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.287359 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.287317    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3514fae-e2c5-4688-8791-adb34c8175c4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.287541 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.287379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2tth\" (UniqueName: \"kubernetes.io/projected/a3514fae-e2c5-4688-8791-adb34c8175c4-kube-api-access-g2tth\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.289869 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.289846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3514fae-e2c5-4688-8791-adb34c8175c4-tls-certs\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.297067 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.297042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2tth\" (UniqueName: \"kubernetes.io/projected/a3514fae-e2c5-4688-8791-adb34c8175c4-kube-api-access-g2tth\") pod \"router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") " pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.382070 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.381991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:43.658504 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:43.658474    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"]
Sep 16 19:56:43.660468 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:56:43.660434    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3514fae_e2c5_4688_8791_adb34c8175c4.slice/crio-857eb958caaa25746b594b3369b8fbc38dfffd9b8aa9827e676ccdc2c54e64b2 WatchSource:0}: Error finding container 857eb958caaa25746b594b3369b8fbc38dfffd9b8aa9827e676ccdc2c54e64b2: Status 404 returned error can't find the container with id 857eb958caaa25746b594b3369b8fbc38dfffd9b8aa9827e676ccdc2c54e64b2
Sep 16 19:56:44.113289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:44.113188    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:56:44.516419 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:44.516380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" event={"ID":"a3514fae-e2c5-4688-8791-adb34c8175c4","Type":"ContainerStarted","Data":"1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373"}
Sep 16 19:56:44.516603 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:44.516431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" event={"ID":"a3514fae-e2c5-4688-8791-adb34c8175c4","Type":"ContainerStarted","Data":"857eb958caaa25746b594b3369b8fbc38dfffd9b8aa9827e676ccdc2c54e64b2"}
Sep 16 19:56:44.516603 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:44.516519    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:56:44.535917 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:44.535854    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podStartSLOduration=1.5358411749999998 podStartE2EDuration="1.535841175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:56:44.534316921 +0000 UTC m=+4388.646113562" watchObservedRunningTime="2026-09-16 19:56:44.535841175 +0000 UTC m=+4388.647637814"
Sep 16 19:56:58.898592 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:56:58.898452    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:15.523072 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:15.523039    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:57:18.938241 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:18.938193    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"]
Sep 16 19:57:19.040052 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:19.040019    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:19.166652 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:19.166617    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:23.243310 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:23.243249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:23.308014 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:23.307983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:57:26.040296 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:26.040255    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:33.958985 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:33.958939    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main" containerID="cri-o://c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e" gracePeriod=45
Sep 16 19:57:36.040855 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:36.040815    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:46.040033 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:46.039990    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:57:46.040537 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:46.040137    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:57:56.039831 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:57:56.039792    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:58:04.216350 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.216321    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:58:04.352121 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.352045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7a64ce-dcd9-430c-9b7b-f2796660979f-tls-certs\") pod \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") "
Sep 16 19:58:04.352121 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.352101    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q84c2\" (UniqueName: \"kubernetes.io/projected/8a7a64ce-dcd9-430c-9b7b-f2796660979f-kube-api-access-q84c2\") pod \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\" (UID: \"8a7a64ce-dcd9-430c-9b7b-f2796660979f\") "
Sep 16 19:58:04.354345 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.354312    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a7a64ce-dcd9-430c-9b7b-f2796660979f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a7a64ce-dcd9-430c-9b7b-f2796660979f" (UID: "8a7a64ce-dcd9-430c-9b7b-f2796660979f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:58:04.354454 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.354352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7a64ce-dcd9-430c-9b7b-f2796660979f-kube-api-access-q84c2" (OuterVolumeSpecName: "kube-api-access-q84c2") pod "8a7a64ce-dcd9-430c-9b7b-f2796660979f" (UID: "8a7a64ce-dcd9-430c-9b7b-f2796660979f"). InnerVolumeSpecName "kube-api-access-q84c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:58:04.453058 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.453029    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a7a64ce-dcd9-430c-9b7b-f2796660979f-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:58:04.453058 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.453057    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q84c2\" (UniqueName: \"kubernetes.io/projected/8a7a64ce-dcd9-430c-9b7b-f2796660979f-kube-api-access-q84c2\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:58:04.931035 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.931001    2566 generic.go:356] "Generic (PLEG): container finished" podID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerID="c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e" exitCode=0
Sep 16 19:58:04.931035 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.931040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" event={"ID":"8a7a64ce-dcd9-430c-9b7b-f2796660979f","Type":"ContainerDied","Data":"c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e"}
Sep 16 19:58:04.931260 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.931051    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"
Sep 16 19:58:04.931260 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.931080    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4" event={"ID":"8a7a64ce-dcd9-430c-9b7b-f2796660979f","Type":"ContainerDied","Data":"6a4b13111cc8d88e38a20dc01bc19bc9a09940237270d455d5c138ec1a81063a"}
Sep 16 19:58:04.931260 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.931103    2566 scope.go:117] "RemoveContainer" containerID="c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e"
Sep 16 19:58:04.940166 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.940137    2566 scope.go:117] "RemoveContainer" containerID="c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e"
Sep 16 19:58:04.940420 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:58:04.940400    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e\": container with ID starting with c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e not found: ID does not exist" containerID="c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e"
Sep 16 19:58:04.940488 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.940431    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e"} err="failed to get container status \"c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e\": rpc error: code = NotFound desc = could not find container \"c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e\": container with ID starting with c230991c5e388df3d79223aa9e7d686cc73c376892734b4a433f7429cb7c673e not found: ID does not exist"
Sep 16 19:58:04.950006 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.949979    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"]
Sep 16 19:58:04.955939 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:04.955910    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-tls-resources-1d9e6e79/tls-verification-test-kserve-router-scheduler-77cd9c86c9-nfml4"]
Sep 16 19:58:06.403022 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:06.402987    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" path="/var/lib/kubelet/pods/8a7a64ce-dcd9-430c-9b7b-f2796660979f/volumes"
Sep 16 19:58:09.003265 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:09.003227    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:09.113433 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:09.113400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:15.669488 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:15.669450    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:19.662739 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.662703    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:19.862824 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.862787    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"]
Sep 16 19:58:19.863180 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.863166    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main"
Sep 16 19:58:19.863234 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.863182    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main"
Sep 16 19:58:19.863270 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.863238    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7a64ce-dcd9-430c-9b7b-f2796660979f" containerName="main"
Sep 16 19:58:19.867895 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.867843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:19.871799 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.871770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"openshift-service-ca.crt\""
Sep 16 19:58:19.872172 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.871847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-epp-sa-dockercfg-j9p79\""
Sep 16 19:58:19.872279 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.871899    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"default-dockercfg-frrmd\""
Sep 16 19:58:19.872342 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.871951    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"kube-root-ca.crt\""
Sep 16 19:58:19.872401 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.871958    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-prestop-hook-22727e61\"/\"prestop-hook-test-kserve-self-signed-certs\""
Sep 16 19:58:19.880074 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.880049    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"]
Sep 16 19:58:19.917975 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.917854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:19.980083 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.980051    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmjk\" (UniqueName: \"kubernetes.io/projected/8d12922b-ca8f-4cb4-9548-6309206b0e16-kube-api-access-vxmjk\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:19.980224 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:19.980093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d12922b-ca8f-4cb4-9548-6309206b0e16-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:20.080897 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:20.080839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmjk\" (UniqueName: \"kubernetes.io/projected/8d12922b-ca8f-4cb4-9548-6309206b0e16-kube-api-access-vxmjk\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:20.080897 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:20.080897    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d12922b-ca8f-4cb4-9548-6309206b0e16-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:20.083402 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:20.083375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d12922b-ca8f-4cb4-9548-6309206b0e16-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:20.090719 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:20.090687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmjk\" (UniqueName: \"kubernetes.io/projected/8d12922b-ca8f-4cb4-9548-6309206b0e16-kube-api-access-vxmjk\") pod \"prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") " pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:20.181543 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:20.181465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:20.455772 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:20.455744    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"]
Sep 16 19:58:20.458059 ip-10-0-143-190 kubenswrapper[2566]: W0916 19:58:20.458026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d12922b_ca8f_4cb4_9548_6309206b0e16.slice/crio-76b4554d9c8320779b5441926537abb65e3fba82497070b8cd18af059739eb1a WatchSource:0}: Error finding container 76b4554d9c8320779b5441926537abb65e3fba82497070b8cd18af059739eb1a: Status 404 returned error can't find the container with id 76b4554d9c8320779b5441926537abb65e3fba82497070b8cd18af059739eb1a
Sep 16 19:58:21.015325 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:21.015293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" event={"ID":"8d12922b-ca8f-4cb4-9548-6309206b0e16","Type":"ContainerStarted","Data":"57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92"}
Sep 16 19:58:21.015854 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:21.015343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" event={"ID":"8d12922b-ca8f-4cb4-9548-6309206b0e16","Type":"ContainerStarted","Data":"76b4554d9c8320779b5441926537abb65e3fba82497070b8cd18af059739eb1a"}
Sep 16 19:58:21.015854 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:21.015380    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:21.035988 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:21.035942    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" podStartSLOduration=2.035927338 podStartE2EDuration="2.035927338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:58:21.03404902 +0000 UTC m=+4485.145845662" watchObservedRunningTime="2026-09-16 19:58:21.035927338 +0000 UTC m=+4485.147723979"
Sep 16 19:58:23.388591 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:23.388558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:23.489370 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:23.489335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:37.306561 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:37.306527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:52.022952 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:52.022917    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 19:58:52.850522 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:52.850487    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:52.872275 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:52.872245    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"]
Sep 16 19:58:52.929075 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:52.929040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:58:55.523299 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:58:55.523255    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:05.523221 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:05.523179    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:07.891570 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:07.891526    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main" containerID="cri-o://1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373" gracePeriod=45
Sep 16 19:59:15.522863 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:15.522820    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:15.523289 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:15.522971    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:59:25.523316 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:25.523271    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:35.522854 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:35.522809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 19:59:38.152305 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.152278    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:59:38.242791 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.242760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3514fae-e2c5-4688-8791-adb34c8175c4-tls-certs\") pod \"a3514fae-e2c5-4688-8791-adb34c8175c4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") "
Sep 16 19:59:38.242967 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.242840    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2tth\" (UniqueName: \"kubernetes.io/projected/a3514fae-e2c5-4688-8791-adb34c8175c4-kube-api-access-g2tth\") pod \"a3514fae-e2c5-4688-8791-adb34c8175c4\" (UID: \"a3514fae-e2c5-4688-8791-adb34c8175c4\") "
Sep 16 19:59:38.244989 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.244963    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3514fae-e2c5-4688-8791-adb34c8175c4-kube-api-access-g2tth" (OuterVolumeSpecName: "kube-api-access-g2tth") pod "a3514fae-e2c5-4688-8791-adb34c8175c4" (UID: "a3514fae-e2c5-4688-8791-adb34c8175c4"). InnerVolumeSpecName "kube-api-access-g2tth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:59:38.245093 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.244989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3514fae-e2c5-4688-8791-adb34c8175c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a3514fae-e2c5-4688-8791-adb34c8175c4" (UID: "a3514fae-e2c5-4688-8791-adb34c8175c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:59:38.343538 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.343480    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a3514fae-e2c5-4688-8791-adb34c8175c4-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:59:38.343538 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.343504    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2tth\" (UniqueName: \"kubernetes.io/projected/a3514fae-e2c5-4688-8791-adb34c8175c4-kube-api-access-g2tth\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 19:59:38.431174 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.431147    2566 generic.go:356] "Generic (PLEG): container finished" podID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerID="1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373" exitCode=0
Sep 16 19:59:38.431287 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.431190    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"
Sep 16 19:59:38.431287 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.431227    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" event={"ID":"a3514fae-e2c5-4688-8791-adb34c8175c4","Type":"ContainerDied","Data":"1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373"}
Sep 16 19:59:38.431287 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.431275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4" event={"ID":"a3514fae-e2c5-4688-8791-adb34c8175c4","Type":"ContainerDied","Data":"857eb958caaa25746b594b3369b8fbc38dfffd9b8aa9827e676ccdc2c54e64b2"}
Sep 16 19:59:38.431426 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.431296    2566 scope.go:117] "RemoveContainer" containerID="1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373"
Sep 16 19:59:38.440062 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.440033    2566 scope.go:117] "RemoveContainer" containerID="1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373"
Sep 16 19:59:38.440336 ip-10-0-143-190 kubenswrapper[2566]: E0916 19:59:38.440316    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373\": container with ID starting with 1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373 not found: ID does not exist" containerID="1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373"
Sep 16 19:59:38.440393 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.440345    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373"} err="failed to get container status \"1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373\": rpc error: code = NotFound desc = could not find container \"1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373\": container with ID starting with 1b736b20dc5c62111ce138dccbf8e6d2090ab0a6c364276e8b0e303e05901373 not found: ID does not exist"
Sep 16 19:59:38.449382 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.449358    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"]
Sep 16 19:59:38.456177 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:38.456156    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-llm-inference-service-b5dd93e6/router-with-refs-pd-test-kserve-router-scheduler-99c668b45zcgf4"]
Sep 16 19:59:40.402163 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:40.402125    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" path="/var/lib/kubelet/pods/a3514fae-e2c5-4688-8791-adb34c8175c4/volumes"
Sep 16 19:59:42.577124 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:42.577073    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:42.669192 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:42.669157    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:46.429235 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:46.429200    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 19:59:47.617482 ip-10-0-143-190 kubenswrapper[2566]: I0916 19:59:47.617421    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:08.843826 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:08.843785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:10.388551 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:10.388511    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:16.273174 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:16.273113    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:21.681333 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:21.681289    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:30.114670 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:30.114630    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:31.063271 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:31.063231    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:31.168586 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:31.168544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:43.305202 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:43.305152    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:43.335466 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:43.335433    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"]
Sep 16 20:00:43.471489 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:43.471460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:00:52.022717 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:52.022674    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:00:58.355424 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:00:58.355381    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main" containerID="cri-o://57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92" gracePeriod=45
Sep 16 20:01:02.022560 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:02.022516    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:12.022708 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:12.022658    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:12.023130 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:12.022774    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 20:01:22.023117 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:22.023073    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:01:28.601662 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.601636    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 20:01:28.656105 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.656077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d12922b-ca8f-4cb4-9548-6309206b0e16-tls-certs\") pod \"8d12922b-ca8f-4cb4-9548-6309206b0e16\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") "
Sep 16 20:01:28.656238 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.656113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxmjk\" (UniqueName: \"kubernetes.io/projected/8d12922b-ca8f-4cb4-9548-6309206b0e16-kube-api-access-vxmjk\") pod \"8d12922b-ca8f-4cb4-9548-6309206b0e16\" (UID: \"8d12922b-ca8f-4cb4-9548-6309206b0e16\") "
Sep 16 20:01:28.658186 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.658155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d12922b-ca8f-4cb4-9548-6309206b0e16-kube-api-access-vxmjk" (OuterVolumeSpecName: "kube-api-access-vxmjk") pod "8d12922b-ca8f-4cb4-9548-6309206b0e16" (UID: "8d12922b-ca8f-4cb4-9548-6309206b0e16"). InnerVolumeSpecName "kube-api-access-vxmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:01:28.658310 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.658192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d12922b-ca8f-4cb4-9548-6309206b0e16-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8d12922b-ca8f-4cb4-9548-6309206b0e16" (UID: "8d12922b-ca8f-4cb4-9548-6309206b0e16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:01:28.757208 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.757168    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d12922b-ca8f-4cb4-9548-6309206b0e16-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 20:01:28.757208 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:28.757205    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmjk\" (UniqueName: \"kubernetes.io/projected/8d12922b-ca8f-4cb4-9548-6309206b0e16-kube-api-access-vxmjk\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 20:01:29.007695 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.007599    2566 generic.go:356] "Generic (PLEG): container finished" podID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerID="57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92" exitCode=0
Sep 16 20:01:29.007695 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.007643    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"
Sep 16 20:01:29.007905 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.007683    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" event={"ID":"8d12922b-ca8f-4cb4-9548-6309206b0e16","Type":"ContainerDied","Data":"57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92"}
Sep 16 20:01:29.007905 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.007746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn" event={"ID":"8d12922b-ca8f-4cb4-9548-6309206b0e16","Type":"ContainerDied","Data":"76b4554d9c8320779b5441926537abb65e3fba82497070b8cd18af059739eb1a"}
Sep 16 20:01:29.007905 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.007768    2566 scope.go:117] "RemoveContainer" containerID="57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92"
Sep 16 20:01:29.017278 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.017247    2566 scope.go:117] "RemoveContainer" containerID="57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92"
Sep 16 20:01:29.017543 ip-10-0-143-190 kubenswrapper[2566]: E0916 20:01:29.017522    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92\": container with ID starting with 57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92 not found: ID does not exist" containerID="57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92"
Sep 16 20:01:29.017614 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.017552    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92"} err="failed to get container status \"57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92\": rpc error: code = NotFound desc = could not find container \"57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92\": container with ID starting with 57e576303e1937b8b540ad3c4df815d61eede5da2134f7711d4334e87cf37f92 not found: ID does not exist"
Sep 16 20:01:29.031766 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.031729    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"]
Sep 16 20:01:29.034187 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:29.034148    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-prestop-hook-22727e61/prestop-hook-test-kserve-router-scheduler-58978fbc99-q7wtn"]
Sep 16 20:01:30.402911 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:30.402860    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" path="/var/lib/kubelet/pods/8d12922b-ca8f-4cb4-9548-6309206b0e16/volumes"
Sep 16 20:01:48.457042 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:48.457002    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:49.755235 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:49.755202    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:51.577180 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:51.577147    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:01:55.119937 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:01:55.119906    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:00.905789 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:00.905727    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:00.976554 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:00.976527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:02.067772 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.067736    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:02.425637 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.425601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:02.642810 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.642765    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"]
Sep 16 20:02:02.643355 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.643333    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main"
Sep 16 20:02:02.643412 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.643361    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main"
Sep 16 20:02:02.643412 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.643374    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main"
Sep 16 20:02:02.643412 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.643383    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main"
Sep 16 20:02:02.643523 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.643490    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3514fae-e2c5-4688-8791-adb34c8175c4" containerName="main"
Sep 16 20:02:02.643523 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.643502    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d12922b-ca8f-4cb4-9548-6309206b0e16" containerName="main"
Sep 16 20:02:02.646950 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.646930    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.649759 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.649737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"kube-root-ca.crt\""
Sep 16 20:02:02.651042 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.651010    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Sep 16 20:02:02.651194 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.651015    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"openshift-service-ca.crt\""
Sep 16 20:02:02.651194 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.651067    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"rolling-upgrade-test-epp-sa-dockercfg-d2jtn\""
Sep 16 20:02:02.651194 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.651160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-test-rolling-upgrade-coordination-a9229638\"/\"default-dockercfg-c294d\""
Sep 16 20:02:02.656849 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.656829    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"]
Sep 16 20:02:02.682174 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.682106    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:02.724978 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.724950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgwz9\" (UniqueName: \"kubernetes.io/projected/14e201bb-7e66-47fa-afaa-aa46422b44d5-kube-api-access-sgwz9\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.725091 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.725075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e201bb-7e66-47fa-afaa-aa46422b44d5-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.825582 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.825550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e201bb-7e66-47fa-afaa-aa46422b44d5-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.825708 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.825594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgwz9\" (UniqueName: \"kubernetes.io/projected/14e201bb-7e66-47fa-afaa-aa46422b44d5-kube-api-access-sgwz9\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.828121 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.828100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e201bb-7e66-47fa-afaa-aa46422b44d5-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.834826 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.834797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgwz9\" (UniqueName: \"kubernetes.io/projected/14e201bb-7e66-47fa-afaa-aa46422b44d5-kube-api-access-sgwz9\") pod \"rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") " pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:02.960221 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:02.960145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:03.233534 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:03.233503    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"]
Sep 16 20:02:03.234641 ip-10-0-143-190 kubenswrapper[2566]: W0916 20:02:03.234601    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e201bb_7e66_47fa_afaa_aa46422b44d5.slice/crio-06abb3f8e1ed5b20bbae6dc5506380fec5864bcf43cd2da0aa80f02fb19f2cf2 WatchSource:0}: Error finding container 06abb3f8e1ed5b20bbae6dc5506380fec5864bcf43cd2da0aa80f02fb19f2cf2: Status 404 returned error can't find the container with id 06abb3f8e1ed5b20bbae6dc5506380fec5864bcf43cd2da0aa80f02fb19f2cf2
Sep 16 20:02:04.188989 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:04.188949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" event={"ID":"14e201bb-7e66-47fa-afaa-aa46422b44d5","Type":"ContainerStarted","Data":"b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e"}
Sep 16 20:02:04.189169 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:04.189018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" event={"ID":"14e201bb-7e66-47fa-afaa-aa46422b44d5","Type":"ContainerStarted","Data":"06abb3f8e1ed5b20bbae6dc5506380fec5864bcf43cd2da0aa80f02fb19f2cf2"}
Sep 16 20:02:04.189169 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:04.189143    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:04.225698 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:04.225646    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podStartSLOduration=2.225633703 podStartE2EDuration="2.225633703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:02:04.224775241 +0000 UTC m=+4708.336571876" watchObservedRunningTime="2026-09-16 20:02:04.225633703 +0000 UTC m=+4708.337430344"
Sep 16 20:02:11.243896 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:11.243832    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:13.853180 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:13.853132    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:30.916046 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:30.916015    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:35.196336 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:35.196300    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:02:37.664089 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:37.664054    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:43.660385 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:43.660353    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:43.761730 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:43.761697    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:57.725600 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:57.725562    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:02:57.828464 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:02:57.828417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:11.459051 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:11.458949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:11.461100 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:11.461070    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"]
Sep 16 20:03:11.551818 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:11.551785    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:03:15.196306 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:15.196264    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:25.196003 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:25.195961    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:26.481389 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:26.481338    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main" containerID="cri-o://b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e" gracePeriod=45
Sep 16 20:03:35.196591 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:35.196551    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:35.197027 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:35.196650    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:03:45.196452 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:45.196411    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:55.196776 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:55.196732    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Sep 16 20:03:56.727424 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.727397    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:03:56.766107 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.766014    2566 generic.go:356] "Generic (PLEG): container finished" podID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerID="b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e" exitCode=0
Sep 16 20:03:56.766107 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.766067    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"
Sep 16 20:03:56.766107 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.766077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" event={"ID":"14e201bb-7e66-47fa-afaa-aa46422b44d5","Type":"ContainerDied","Data":"b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e"}
Sep 16 20:03:56.766375 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.766166    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb" event={"ID":"14e201bb-7e66-47fa-afaa-aa46422b44d5","Type":"ContainerDied","Data":"06abb3f8e1ed5b20bbae6dc5506380fec5864bcf43cd2da0aa80f02fb19f2cf2"}
Sep 16 20:03:56.766375 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.766197    2566 scope.go:117] "RemoveContainer" containerID="b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e"
Sep 16 20:03:56.774963 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.774945    2566 scope.go:117] "RemoveContainer" containerID="b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e"
Sep 16 20:03:56.775281 ip-10-0-143-190 kubenswrapper[2566]: E0916 20:03:56.775254    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e\": container with ID starting with b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e not found: ID does not exist" containerID="b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e"
Sep 16 20:03:56.775350 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.775292    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e"} err="failed to get container status \"b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e\": rpc error: code = NotFound desc = could not find container \"b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e\": container with ID starting with b99829bbf4f56ee6b463ead9907dc2ab24cbcc15d9a5a524897e8b9355ac425e not found: ID does not exist"
Sep 16 20:03:56.824714 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.824685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e201bb-7e66-47fa-afaa-aa46422b44d5-tls-certs\") pod \"14e201bb-7e66-47fa-afaa-aa46422b44d5\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") "
Sep 16 20:03:56.824845 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.824742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgwz9\" (UniqueName: \"kubernetes.io/projected/14e201bb-7e66-47fa-afaa-aa46422b44d5-kube-api-access-sgwz9\") pod \"14e201bb-7e66-47fa-afaa-aa46422b44d5\" (UID: \"14e201bb-7e66-47fa-afaa-aa46422b44d5\") "
Sep 16 20:03:56.827016 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.826982    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e201bb-7e66-47fa-afaa-aa46422b44d5-kube-api-access-sgwz9" (OuterVolumeSpecName: "kube-api-access-sgwz9") pod "14e201bb-7e66-47fa-afaa-aa46422b44d5" (UID: "14e201bb-7e66-47fa-afaa-aa46422b44d5"). InnerVolumeSpecName "kube-api-access-sgwz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 20:03:56.827125 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.827021    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e201bb-7e66-47fa-afaa-aa46422b44d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "14e201bb-7e66-47fa-afaa-aa46422b44d5" (UID: "14e201bb-7e66-47fa-afaa-aa46422b44d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 20:03:56.925296 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.925248    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgwz9\" (UniqueName: \"kubernetes.io/projected/14e201bb-7e66-47fa-afaa-aa46422b44d5-kube-api-access-sgwz9\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 20:03:56.925296 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:56.925290    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14e201bb-7e66-47fa-afaa-aa46422b44d5-tls-certs\") on node \"ip-10-0-143-190.ec2.internal\" DevicePath \"\""
Sep 16 20:03:57.089987 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:57.089948    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"]
Sep 16 20:03:57.094359 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:57.094330    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-test-rolling-upgrade-coordination-a9229638/rolling-upgrade-test-kserve-router-scheduler-74c7697f55-r4hdb"]
Sep 16 20:03:58.402267 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:58.402231    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" path="/var/lib/kubelet/pods/14e201bb-7e66-47fa-afaa-aa46422b44d5/volumes"
Sep 16 20:03:59.340525 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:03:59.340485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:00.400450 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:00.400386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:00.502900 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:00.502845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:05.547395 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:05.547363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:11.569366 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:11.569318    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:11.674341 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:11.674306    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:04:23.696902 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:23.696798    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:04:23.891618 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:23.891584    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:04:24.672842 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:24.672809    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:04:24.862494 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:24.862464    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:04:25.747974 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:25.747931    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:04:25.945046 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:25.945014    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:04:26.696216 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:26.696182    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:04:26.902793 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:26.902763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:04:28.631646 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:28.631556    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:28.667902 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:28.667866    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:30.528349 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:30.528305    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:30.600301 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:30.600268    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:32.482927 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:32.482890    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:32.522675 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:32.522628    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:34.201391 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:34.201354    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:34.351918 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:34.351874    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:35.947596 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:35.947559    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:35.982148 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:35.982114    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:37.610281 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:37.610245    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:37.666081 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:37.666042    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:39.284748 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:39.284707    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:39.322319 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:39.322287    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:40.960953 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:40.960915    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:41.081612 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:41.081574    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:42.836286 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:42.836240    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:42.885775 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:42.885732    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:44.474628 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:44.474590    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:44.507475 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:44.507443    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:46.075736 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:46.075699    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:46.119546 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:46.119509    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:47.796268 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:47.796217    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:47.828849 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:47.828821    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:49.473517 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:49.473474    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:49.509324 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:49.509277    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:51.081071 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:51.081035    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:51.117179 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:51.117155    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:52.717156 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:52.717118    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:52.749510 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:52.749472    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:54.422198 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:54.422162    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:54.474636 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:54.474607    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:56.101474 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:56.101444    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:56.137062 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:56.137032    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:57.793698 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:57.793658    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:57.849858 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:57.849823    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:04:59.505077 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:59.505033    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:04:59.545547 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:04:59.545507    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:01.187086 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:01.187049    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:01.222949 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:01.222907    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:02.820930 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:02.820890    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:02.875791 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:02.875752    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:04.598826 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:04.598792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:04.649066 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:04.649032    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:06.230724 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:06.230680    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:06.276386 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:06.276355    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:07.866698 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:07.866642    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:07.898734 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:07.898702    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:09.489677 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:09.489640    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:09.524784 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:09.524749    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:11.552135 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.552102    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:11.672199 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.672161    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qnxnw/must-gather-vpcjc"]
Sep 16 20:05:11.672517 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.672504    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main"
Sep 16 20:05:11.672562 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.672518    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main"
Sep 16 20:05:11.672599 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.672587    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e201bb-7e66-47fa-afaa-aa46422b44d5" containerName="main"
Sep 16 20:05:11.675780 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.675754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.678772 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.678749    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qnxnw\"/\"default-dockercfg-kqdgr\""
Sep 16 20:05:11.679130 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.679110    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnxnw\"/\"kube-root-ca.crt\""
Sep 16 20:05:11.679234 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.679185    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qnxnw\"/\"openshift-service-ca.crt\""
Sep 16 20:05:11.687902 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.687856    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/must-gather-vpcjc"]
Sep 16 20:05:11.702122 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.702088    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:11.722275 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.722245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8g2\" (UniqueName: \"kubernetes.io/projected/e4ada6f0-e406-4327-9f8c-aa7db784bcba-kube-api-access-px8g2\") pod \"must-gather-vpcjc\" (UID: \"e4ada6f0-e406-4327-9f8c-aa7db784bcba\") " pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.722433 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.722383    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4ada6f0-e406-4327-9f8c-aa7db784bcba-must-gather-output\") pod \"must-gather-vpcjc\" (UID: \"e4ada6f0-e406-4327-9f8c-aa7db784bcba\") " pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.799979 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.799946    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:11.823790 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.823692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4ada6f0-e406-4327-9f8c-aa7db784bcba-must-gather-output\") pod \"must-gather-vpcjc\" (UID: \"e4ada6f0-e406-4327-9f8c-aa7db784bcba\") " pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.823790 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.823740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8g2\" (UniqueName: \"kubernetes.io/projected/e4ada6f0-e406-4327-9f8c-aa7db784bcba-kube-api-access-px8g2\") pod \"must-gather-vpcjc\" (UID: \"e4ada6f0-e406-4327-9f8c-aa7db784bcba\") " pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.824082 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.824061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e4ada6f0-e406-4327-9f8c-aa7db784bcba-must-gather-output\") pod \"must-gather-vpcjc\" (UID: \"e4ada6f0-e406-4327-9f8c-aa7db784bcba\") " pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.832528 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.832497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8g2\" (UniqueName: \"kubernetes.io/projected/e4ada6f0-e406-4327-9f8c-aa7db784bcba-kube-api-access-px8g2\") pod \"must-gather-vpcjc\" (UID: \"e4ada6f0-e406-4327-9f8c-aa7db784bcba\") " pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:11.985296 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:11.985248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/must-gather-vpcjc"
Sep 16 20:05:12.290125 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:12.290093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/must-gather-vpcjc"]
Sep 16 20:05:12.292186 ip-10-0-143-190 kubenswrapper[2566]: W0916 20:05:12.292155    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4ada6f0_e406_4327_9f8c_aa7db784bcba.slice/crio-3289749c53713d996cd33db688d32a375e98b711c890ad2849ad19372a2c355f WatchSource:0}: Error finding container 3289749c53713d996cd33db688d32a375e98b711c890ad2849ad19372a2c355f: Status 404 returned error can't find the container with id 3289749c53713d996cd33db688d32a375e98b711c890ad2849ad19372a2c355f
Sep 16 20:05:12.294094 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:12.294075    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 20:05:13.169115 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:13.169088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/must-gather-vpcjc" event={"ID":"e4ada6f0-e406-4327-9f8c-aa7db784bcba","Type":"ContainerStarted","Data":"ae2dd7f5512785a1865dcffd903a01e600277476fb34369b069d13081e8c0501"}
Sep 16 20:05:13.169456 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:13.169136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/must-gather-vpcjc" event={"ID":"e4ada6f0-e406-4327-9f8c-aa7db784bcba","Type":"ContainerStarted","Data":"3289749c53713d996cd33db688d32a375e98b711c890ad2849ad19372a2c355f"}
Sep 16 20:05:14.176765 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:14.176716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/must-gather-vpcjc" event={"ID":"e4ada6f0-e406-4327-9f8c-aa7db784bcba","Type":"ContainerStarted","Data":"76d940dfb9fe4a30b09d02af542b44e2b7479d4b531b3f381b65d910293522f2"}
Sep 16 20:05:14.201057 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:14.200985    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qnxnw/must-gather-vpcjc" podStartSLOduration=2.426807861 podStartE2EDuration="3.200963304s" totalImagesPullingTime="774.155443ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 20:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:14.196911863 +0000 UTC m=+4898.308708506" watchObservedRunningTime="2026-09-16 20:05:14.200963304 +0000 UTC m=+4898.312759947"
Sep 16 20:05:14.696673 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:14.696629    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-42hjm_38006a9b-3fb4-4888-8733-f4468724b1e5/global-pull-secret-syncer/0.log"
Sep 16 20:05:14.824361 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:14.824321    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log"
Sep 16 20:05:15.021751 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:15.021673    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-190.ec2.internal_eac4e7fa29b5d05ce71ef20ddfb25659/haproxy/0.log"
Sep 16 20:05:18.117595 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:18.117526    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:18.213693 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:18.213636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbrg4_187276b7-26ce-4ea2-9709-359d01b580d9/konnectivity-agent/0.log" limit=1024
Sep 16 20:05:20.234913 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:20.234748    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-6d4cb79c-f4lrw_9ff329cd-9941-49e0-aaee-8f2467221ed6/manager/0.log"
Sep 16 20:05:20.332589 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:20.332488    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5ffb4b4ccb-8tx67_97e01e16-c64f-4213-88e3-158692641640/manager/0.log"
Sep 16 20:05:21.609420 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.609374    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/alertmanager/0.log"
Sep 16 20:05:21.639667 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.639631    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/config-reloader/0.log"
Sep 16 20:05:21.665741 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.665705    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/kube-rbac-proxy-web/0.log"
Sep 16 20:05:21.694231 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.694202    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/kube-rbac-proxy/0.log"
Sep 16 20:05:21.730291 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.730263    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/kube-rbac-proxy-metric/0.log"
Sep 16 20:05:21.761533 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.761499    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/prom-label-proxy/0.log"
Sep 16 20:05:21.793967 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:21.793920    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b4fb1523-841c-45aa-a101-2c2612d9b1d6/init-config-reloader/0.log"
Sep 16 20:05:22.055294 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:22.055255    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-z8kgl_76833619-3c79-4c52-aad7-5efe9d093d61/monitoring-plugin/0.log"
Sep 16 20:05:22.349079 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:22.348994    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75kdj_50c34b21-5390-408d-a458-896b801676cf/node-exporter/0.log"
Sep 16 20:05:22.456764 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:22.456726    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75kdj_50c34b21-5390-408d-a458-896b801676cf/kube-rbac-proxy/0.log"
Sep 16 20:05:22.515409 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:22.515379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-75kdj_50c34b21-5390-408d-a458-896b801676cf/init-textfile/0.log"
Sep 16 20:05:23.242320 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:23.242290    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-44kmx_70443ead-2fd9-46cc-b064-e4f63b3cefbf/prometheus-operator-admission-webhook/0.log"
Sep 16 20:05:24.565727 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.565681    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"]
Sep 16 20:05:24.571174 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.571139    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.574576 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.574546    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"]
Sep 16 20:05:24.644549 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.644502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcgfg\" (UniqueName: \"kubernetes.io/projected/0ba2ab60-ae6c-4d85-8bf1-618154e92537-kube-api-access-pcgfg\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.644924 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.644896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-sys\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.645119 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.645099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-lib-modules\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.645265 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.645250    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-podres\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.645552 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.645533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-proc\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746569 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-proc\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746569 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcgfg\" (UniqueName: \"kubernetes.io/projected/0ba2ab60-ae6c-4d85-8bf1-618154e92537-kube-api-access-pcgfg\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746787 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-proc\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746787 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-sys\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746787 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-sys\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746787 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-lib-modules\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746999 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-podres\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.746999 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746872    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-lib-modules\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.747070 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.746999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0ba2ab60-ae6c-4d85-8bf1-618154e92537-podres\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.755330 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.755297    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcgfg\" (UniqueName: \"kubernetes.io/projected/0ba2ab60-ae6c-4d85-8bf1-618154e92537-kube-api-access-pcgfg\") pod \"perf-node-gather-daemonset-6xl7x\" (UID: \"0ba2ab60-ae6c-4d85-8bf1-618154e92537\") " pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:24.888302 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:24.888263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:25.195086 ip-10-0-143-190 kubenswrapper[2566]: W0916 20:05:25.195049    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0ba2ab60_ae6c_4d85_8bf1_618154e92537.slice/crio-f3856a0f95134dc16d645a1a9b2bc25532d786aaf871c4f1eb8312fd86cde242 WatchSource:0}: Error finding container f3856a0f95134dc16d645a1a9b2bc25532d786aaf871c4f1eb8312fd86cde242: Status 404 returned error can't find the container with id f3856a0f95134dc16d645a1a9b2bc25532d786aaf871c4f1eb8312fd86cde242
Sep 16 20:05:25.195831 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:25.195751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"]
Sep 16 20:05:25.263329 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:25.263289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x" event={"ID":"0ba2ab60-ae6c-4d85-8bf1-618154e92537","Type":"ContainerStarted","Data":"f3856a0f95134dc16d645a1a9b2bc25532d786aaf871c4f1eb8312fd86cde242"}
Sep 16 20:05:25.607663 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:25.607569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-bdc6bbd-p7q24_d0e36b21-a5e2-4100-b41a-9ab8e66a79b0/console/0.log"
Sep 16 20:05:25.666682 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:25.666646    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-nzwvv_37a84283-7872-4a59-9f10-fce8fb541c19/download-server/0.log"
Sep 16 20:05:26.273238 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:26.273194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x" event={"ID":"0ba2ab60-ae6c-4d85-8bf1-618154e92537","Type":"ContainerStarted","Data":"e9eae1277b924df01fa12a6375fbf0a369b1c2d39d3852e2f9cb638bec7c5bb4"}
Sep 16 20:05:26.273453 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:26.273276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:26.305804 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:26.305742    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x" podStartSLOduration=2.305722376 podStartE2EDuration="2.305722376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 20:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 20:05:26.305142707 +0000 UTC m=+4910.416939348" watchObservedRunningTime="2026-09-16 20:05:26.305722376 +0000 UTC m=+4910.417519067"
Sep 16 20:05:27.513500 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:27.513464    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94j8c_51a79dc4-f304-4b8d-a892-f3836095cfe0/dns/0.log"
Sep 16 20:05:27.544208 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:27.544172    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-94j8c_51a79dc4-f304-4b8d-a892-f3836095cfe0/kube-rbac-proxy/0.log"
Sep 16 20:05:27.679687 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:27.679647    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-s22pq_f7ac3ad0-6e65-477d-8498-641c46de4e73/dns-node-resolver/0.log"
Sep 16 20:05:28.363515 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:28.363483    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59665b5c65-ht8k9_3ac21c56-2d25-4921-87c3-16c2485093c1/registry/0.log"
Sep 16 20:05:28.424036 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:28.424003    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bbdc7_0f1787e6-b472-407d-8946-9d975b058d80/node-ca/0.log"
Sep 16 20:05:29.561665 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:29.561619    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:05:29.806996 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:29.806957    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:05:30.491109 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:30.491079    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4r4ns_d7c0e28f-7c81-4d9b-8ca4-0d3fd594ddd6/serve-healthcheck-canary/0.log"
Sep 16 20:05:31.203169 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:31.203138    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xqjz5_6f124bba-8be5-4999-86ba-40e295e86078/kube-rbac-proxy/0.log"
Sep 16 20:05:31.238637 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:31.238605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xqjz5_6f124bba-8be5-4999-86ba-40e295e86078/exporter/0.log"
Sep 16 20:05:31.268509 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:31.268478    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xqjz5_6f124bba-8be5-4999-86ba-40e295e86078/extractor/0.log"
Sep 16 20:05:32.289721 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:32.289688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qnxnw/perf-node-gather-daemonset-6xl7x"
Sep 16 20:05:34.918425 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:34.918363    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:05:35.191944 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:35.191905    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:05:35.490932 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:35.490838    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-f8c46b677-cjr45_2409e2cf-a6d0-43e2-b63f-5fe61d6f6e78/manager/0.log"
Sep 16 20:05:36.071843 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:36.071791    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-c98cf57d5-wmmtx_a3c72322-36a0-4841-aeb4-e6156d441e1f/manager/0.log"
Sep 16 20:05:36.448015 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:36.447451    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/1.log"
Sep 16 20:05:36.869922 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:36.869871    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-mkxxm_78a6b5a3-3645-4693-a957-3354e0c592ca/discovery/0.log"
Sep 16 20:05:37.543677 ip-10-0-143-190 kubenswrapper[2566]: I0916 20:05:37.543584    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-rr9qw_f2d8b15d-8a41-4b4d-ac24-36c411cc6a60/manager/0.log"