Sep 08 14:53:18.932486 ip-10-0-136-157 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 08 14:53:18.932501 ip-10-0-136-157 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 08 14:53:18.932510 ip-10-0-136-157 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 08 14:53:18.932785 ip-10-0-136-157 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 08 14:53:29.147020 ip-10-0-136-157 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 08 14:53:29.147035 ip-10-0-136-157 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 4853ab396afd4512aed612916e32c2d7 --
Sep 08 14:55:21.236212 ip-10-0-136-157 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 14:55:21.641497 ip-10-0-136-157 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 14:55:21.641497 ip-10-0-136-157 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 14:55:21.641497 ip-10-0-136-157 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 14:55:21.641497 ip-10-0-136-157 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 14:55:21.641497 ip-10-0-136-157 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 14:55:21.642774 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.642672    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 14:55:21.648338 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648318    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 14:55:21.648338 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648336    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 14:55:21.648338 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648340    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 14:55:21.648338 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648344    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648347    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648351    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648354    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648357    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648359    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648362    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648364    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648367    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648370    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648372    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648375    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648378    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648380    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648382    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648385    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648387    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648390    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648392    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648395    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 14:55:21.648497 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648397    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648400    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648403    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648405    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648408    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648410    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648415    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648419    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648422    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648425    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648430    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648432    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648435    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648437    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648441    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648444    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648448    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648451    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648454    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 14:55:21.648977 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648458    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648460    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648463    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648466    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648469    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648472    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648474    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648477    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648479    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648482    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648484    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648487    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648490    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648492    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648495    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648497    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648500    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648503    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648505    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648508    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 14:55:21.649417 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648510    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648513    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648515    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648518    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648531    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648534    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648537    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648542    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648544    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648547    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648549    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648552    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648555    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648557    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648560    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648562    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648564    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648567    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648569    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648572    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 14:55:21.649899 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648575    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648579    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648582    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648585    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648587    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648590    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648592    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648594    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648597    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648599    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648602    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648604    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648606    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648609    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648612    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648615    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.648618    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649396    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649403    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649407    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 14:55:21.650375 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649410    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649413    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649416    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649419    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649422    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649425    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649428    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649430    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649434    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649437    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649440    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649442    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649445    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649447    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649451    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649453    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649456    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649459    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649461    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 14:55:21.650894 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649464    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649466    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649468    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649471    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649473    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649476    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649478    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649481    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649484    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649486    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649488    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649491    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649494    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649496    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649498    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649501    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649503    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649506    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649508    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649511    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 14:55:21.651372 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649513    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649516    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649518    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649542    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649546    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649550    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649554    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649556    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649560    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649564    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649568    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649571    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649573    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649576    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649578    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649581    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649583    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649586    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649589    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 14:55:21.651864 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649591    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649594    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649596    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649599    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649601    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649604    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649607    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649610    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649612    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649614    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649617    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649620    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649623    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649626    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649629    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649631    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649634    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649638    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649640    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 14:55:21.652326 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649643    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649646    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649648    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649651    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649654    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649656    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649658    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649661    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649663    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649665    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649668    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649670    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649672    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649675    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649677    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649680    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649682    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649685    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.649688    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650066    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 14:55:21.652791 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650071    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650074    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650077    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650079    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650082    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650085    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650087    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650090    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650092    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650095    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650097    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650100    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650102    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650104    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650107    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650111    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650115    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650117    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650120    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650122    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 14:55:21.653266 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650124    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650127    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650129    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650132    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650134    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650136    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650140    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650142    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650145    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650147    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650150    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650152    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650155    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650157    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650160    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650163    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650165    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650167    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650170    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650172    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 14:55:21.653744 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650176    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650178    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650180    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650183    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650185    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650188    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650193    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650195    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650198    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650201    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650203    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650206    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650208    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650210    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650213    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650215    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650217    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650220    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650222    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650225    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 14:55:21.654198 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650227    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650231    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650234    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650237    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650240    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650243    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650252    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650255    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650257    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650259    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650262    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650264    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650267    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650269    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650271    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650274    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650276    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650278    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650282    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 14:55:21.654676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650285    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650287    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650290    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650293    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650295    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650299    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650302    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650304    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650307    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650310    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650313    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650315    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650317    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650320    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650322    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650324    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650327    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650329    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.650331    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650406    2578 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650439    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 14:55:21.655166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650445    2578 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650449    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650453    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650456    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650460    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650464    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650467    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650469    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650473    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650476    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650479    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650482    2578 flags.go:64] FLAG: --cgroup-root=""
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650485    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650487    2578 flags.go:64] FLAG: --client-ca-file=""
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650491    2578 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650494    2578 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650497    2578 flags.go:64] FLAG: --cluster-domain=""
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650500    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650503    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650506    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650509    2578 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650513    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650515    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 14:55:21.655707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650518    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650535    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650538    2578 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650541    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650544    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650547    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650550    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650553    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650556    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650559    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650562    2578 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650564    2578 flags.go:64] FLAG: --enable-server="true"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650567    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650573    2578 flags.go:64] FLAG: --event-burst="100"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650576    2578 flags.go:64] FLAG: --event-qps="50"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650578    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650581    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650584    2578 flags.go:64] FLAG: --eviction-hard=""
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650592    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650596    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650599    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650601    2578 flags.go:64] FLAG: --eviction-soft=""
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650605    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650608    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650611    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 14:55:21.656242 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650614    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650617    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650620    2578 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650622    2578 flags.go:64] FLAG: --feature-gates=""
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650630    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650633    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650636    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650639    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650642    2578 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650644    2578 flags.go:64] FLAG: --help="false"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650647    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650650    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650653    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650656    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650659    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650662    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650665    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650667    2578 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650670    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650672    2578 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650675    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650678    2578 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650681    2578 flags.go:64] FLAG: --kube-reserved=""
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650683    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 14:55:21.656865 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650686    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650689    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650691    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650694    2578 flags.go:64] FLAG: --lock-file=""
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650701    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650704    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650707    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650712    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650715    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650718    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650721    2578 flags.go:64] FLAG: --logging-format="text"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650723    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650726    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650729    2578 flags.go:64] FLAG: --manifest-url=""
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650732    2578 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650736    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650739    2578 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650742    2578 flags.go:64] FLAG: --max-pods="110"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650745    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650748    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650750    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650753    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650756    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650758    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650761    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 14:55:21.657460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650768    2578 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650771    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650774    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650777    2578 flags.go:64] FLAG: --pod-cidr=""
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650779    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650785    2578 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650788    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650791    2578 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650794    2578 flags.go:64] FLAG: --port="10250"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650797    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650800    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ddf6cbe24c502d06"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650803    2578 flags.go:64] FLAG: --qos-reserved=""
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650807    2578 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650810    2578 flags.go:64] FLAG: --register-node="true"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650813    2578 flags.go:64] FLAG: --register-with-taints=""
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650816    2578 flags.go:64] FLAG: --registry-burst="10"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650820    2578 flags.go:64] FLAG: --registry-qps="5"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650822    2578 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650825    2578 flags.go:64] FLAG: --reserved-memory=""
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650829    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650832    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650834    2578 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650837    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650840    2578 flags.go:64] FLAG: --runonce="false"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650842    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 14:55:21.658073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650845    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650848    2578 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650850    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650853    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650856    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650859    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650862    2578 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650865    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650867    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650870    2578 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650873    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650875    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650878    2578 flags.go:64] FLAG: --system-cgroups=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650881    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650886    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650888    2578 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650892    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650896    2578 flags.go:64] FLAG: --tls-min-version=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650899    2578 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650903    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650906    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650909    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650911    2578 flags.go:64] FLAG: --v="2"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650915    2578 flags.go:64] FLAG: --version="false"
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650922    2578 flags.go:64] FLAG: --vmodule=""
Sep 08 14:55:21.658666 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650926    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.650929    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651077    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651081    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651084    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651087    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651089    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651092    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651094    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651097    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651099    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651103    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651106    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651109    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651112    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651115    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651117    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651120    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651123    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651127    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 14:55:21.659379 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651130    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651133    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651135    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651139    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651142    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651144    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651148    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651151    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651154    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651156    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651159    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651161    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651164    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651167    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651169    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651172    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651175    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651177    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651180    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651183    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 14:55:21.659872 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651185    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651188    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651190    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651193    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651196    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651198    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651201    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651203    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651205    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651208    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651210    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651213    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651215    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651218    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651220    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651224    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651226    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651228    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651232    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651235    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 14:55:21.660346 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651237    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651240    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651242    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651244    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651247    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651250    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651252    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651254    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651257    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651260    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651262    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651265    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651267    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651269    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651272    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651274    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651277    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651279    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651281    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651284    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 14:55:21.660849 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651286    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651289    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651291    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651293    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651297    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651300    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651303    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651307    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651309    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651312    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651316    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651319    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651321    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651324    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651326    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651329    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651331    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651334    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651336    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651339    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 14:55:21.661318 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.651341    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.651932    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.659671    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.659686    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.660539    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.660546    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660610    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660614    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660618    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660621    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660623    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660626    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660629    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660631    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660634    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 14:55:21.661856 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660636    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660638    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660641    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660645    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660649    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660654    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660657    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660660    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660663    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660666    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660668    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660671    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660674    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660677    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660680    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660682    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660685    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660688    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 14:55:21.662252 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660690    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660693    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660697    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660699    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660702    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660705    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660708    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660710    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660712    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660715    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660717    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660720    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660722    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660724    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660727    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660729    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660731    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660734    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660736    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660739    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 14:55:21.662676 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660741    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660744    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660746    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660749    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660751    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660754    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660757    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660759    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660762    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660765    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660767    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660770    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660772    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660775    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660777    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660783    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660785    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660789    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660792    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 14:55:21.663141 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660795    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660798    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660800    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660803    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660806    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660808    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660811    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660814    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660816    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660819    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660821    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660823    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660826    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660828    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660831    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660833    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660835    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660838    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660841    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660843    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 14:55:21.663595 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660846    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660848    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660851    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660854    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660856    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660859    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660861    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660864    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660866    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660874    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660877    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660879    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.660882    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.660887    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661017    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 14:55:21.664057 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661022    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661026    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661029    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661033    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661035    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661038    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661040    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661043    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661046    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661048    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661050    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661053    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661055    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661057    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661060    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661062    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661065    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661067    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661070    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 14:55:21.664406 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661073    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661075    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661078    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661080    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661082    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661085    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661087    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661090    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661092    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661095    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661098    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661100    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661102    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661105    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661107    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661109    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661112    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661114    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661117    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 14:55:21.664855 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661119    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661121    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661124    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661126    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661129    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661131    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661133    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661135    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661138    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661140    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661143    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661145    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661148    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661150    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661152    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661155    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661157    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661159    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661162    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661164    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 14:55:21.665285 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661166    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661169    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661172    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661174    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661176    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661179    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661182    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661184    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661187    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661189    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661192    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661194    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661196    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661199    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661201    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661203    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661206    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661208    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661210    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661212    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 14:55:21.665760 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661215    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661218    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661220    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661222    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661225    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661227    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661230    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661232    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661234    2578 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661237    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661239    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661241    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661245    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661248    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661251    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661254    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661257    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661259    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661262    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 14:55:21.666211 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:21.661264    2578 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 14:55:21.666653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.661269    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 14:55:21.666653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.661356    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 14:55:21.666653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.663187    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 14:55:21.666653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.664230    2578 server.go:1019] "Starting client certificate rotation"
Sep 08 14:55:21.666653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.664327    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 14:55:21.666653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.664363    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 14:55:21.667183 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.667170    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 14:55:21.671321 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.671281    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 14:55:21.680663 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.680646    2578 log.go:25] "Validated CRI v1 runtime API"
Sep 08 14:55:21.683910 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.683895    2578 log.go:25] "Validated CRI v1 image API"
Sep 08 14:55:21.684875 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.684862    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 14:55:21.688455 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.688427    2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80c634c4-2a9b-4750-9f41-037e3321a2cf:/dev/nvme0n1p4 99ec2804-c9d1-40ac-8bd4-50fd7b615abc:/dev/nvme0n1p3]
Sep 08 14:55:21.688515 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.688454    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 14:55:21.694677 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.694573    2578 manager.go:216] Machine: {Timestamp:2026-09-08 14:55:21.692862913 +0000 UTC m=+0.344287213 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199437 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bd1a70bbd0c5077ba57b27fa19a90 SystemUUID:ec2bd1a7-0bbd-0c50-77ba-57b27fa19a90 BootID:4853ab39-6afd-4512-aed6-12916e32c2d7 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:61:c6:d6:9a:c1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:61:c6:d6:9a:c1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:25:d2:d3:3e:b2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 14:55:21.694677 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.694673    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 14:55:21.694777 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.694771    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 14:55:21.697997 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.697980    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 14:55:21.698286 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.698262    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 14:55:21.698409 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.698290    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 14:55:21.699604 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.699594    2578 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 14:55:21.699640 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.699608    2578 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 14:55:21.699640 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.699615    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 14:55:21.700215 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.700205    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 14:55:21.701379 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.701368    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 14:55:21.704945 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.704934    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 14:55:21.705056 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705048    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 14:55:21.705087 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705075    2578 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 14:55:21.705087 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705084    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 14:55:21.705138 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705099    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 14:55:21.705138 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705107    2578 kubelet.go:398] "Adding apiserver pod source"
Sep 08 14:55:21.705138 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705116    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 14:55:21.705498 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705481    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 14:55:21.705570 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.705505    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 14:55:21.708920 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.708884    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 14:55:21.712460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712436    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 14:55:21.712564 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712486    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 14:55:21.712691 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712678    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 14:55:21.712750 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712708    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 14:55:21.712750 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712717    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 14:55:21.712750 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712726    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 14:55:21.712750 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712739    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 14:55:21.712750 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.712749    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 14:55:21.713431 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.713412    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 14:55:21.713506 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.713438    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 14:55:21.713506 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.713450    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 14:55:21.713506 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.713457    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 14:55:21.713506 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.713470    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 14:55:21.713506 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.713480    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 14:55:21.714926 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.714909    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 14:55:21.714926 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.714925    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 14:55:21.716086 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.716064    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 14:55:21.716176 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.716065    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 14:55:21.717857 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.717839    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 14:55:21.719417 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.719403    2578 server.go:1268] "Started kubelet"
Sep 08 14:55:21.719534 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.719463    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 14:55:21.719587 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.719506    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 14:55:21.719617 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.719603    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 14:55:21.720274 ip-10-0-136-157 systemd[1]: Started Kubernetes Kubelet.
Sep 08 14:55:21.721125 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.721030    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 14:55:21.721434 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.721390    2578 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 14:55:21.723754 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.723734    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vpzz8"
Sep 08 14:55:21.725504 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.725485    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 14:55:21.726171 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.725254    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-157.ec2.internal.18d36043e5200cf5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-157.ec2.internal,UID:ip-10-0-136-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-157.ec2.internal,},FirstTimestamp:2026-09-08 14:55:21.719377141 +0000 UTC m=+0.370801440,LastTimestamp:2026-09-08 14:55:21.719377141 +0000 UTC m=+0.370801440,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-157.ec2.internal,}"
Sep 08 14:55:21.726276 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.726187    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 14:55:21.726276 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.726249    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 14:55:21.726366 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.726342    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:21.726920 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.726882    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 14:55:21.726920 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.726901    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 14:55:21.727073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.726883    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 14:55:21.727073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727047    2578 factory.go:55] Registering systemd factory
Sep 08 14:55:21.727073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727057    2578 factory.go:223] Registration of the systemd container factory successfully
Sep 08 14:55:21.727073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727058    2578 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 14:55:21.727073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727068    2578 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 14:55:21.727212 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727199    2578 factory.go:153] Registering CRI-O factory
Sep 08 14:55:21.727212 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727205    2578 factory.go:223] Registration of the crio container factory successfully
Sep 08 14:55:21.727269 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727241    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 14:55:21.727269 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727254    2578 factory.go:103] Registering Raw factory
Sep 08 14:55:21.727269 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727263    2578 manager.go:1196] Started watching for new ooms in manager
Sep 08 14:55:21.727701 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.727685    2578 manager.go:349] Starting recovery of all containers
Sep 08 14:55:21.734787 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.734761    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 14:55:21.734888 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.734869    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 14:55:21.735102 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.735085    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vpzz8"
Sep 08 14:55:21.738268 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.738246    2578 manager.go:354] Recovery completed
Sep 08 14:55:21.743536 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.743509    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.745259 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745242    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.745320 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745269    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.745320 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745279    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.745798 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745785    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 14:55:21.745798 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745796    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.745885 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745800    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 14:55:21.745885 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.745815    2578 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 14:55:21.746471 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.746457    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.746513 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.746483    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.746513 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.746495    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.747361 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.747291    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-157.ec2.internal.18d36043e6aaf22c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-157.ec2.internal,UID:ip-10-0-136-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-157.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-157.ec2.internal,},FirstTimestamp:2026-09-08 14:55:21.745257004 +0000 UTC m=+0.396681306,LastTimestamp:2026-09-08 14:55:21.745257004 +0000 UTC m=+0.396681306,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-157.ec2.internal,}"
Sep 08 14:55:21.747751 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.747739    2578 policy_none.go:49] "None policy: Start"
Sep 08 14:55:21.747785 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.747756    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 14:55:21.747785 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.747765    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 14:55:21.749708 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.749698    2578 policy_none.go:47] "Start"
Sep 08 14:55:21.792046 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.792034    2578 manager.go:337] "Starting Device Plugin manager"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.792066    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.792075    2578 server.go:87] "Starting device plugin registration server"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.792290    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.792303    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.792997    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.793015    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.793052    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.793085    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 14:55:21.806586 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.793091    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 14:55:21.824107 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.824078    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 14:55:21.825317 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.825295    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 14:55:21.825387 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.825340    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 14:55:21.825387 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.825360    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 14:55:21.825481 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.825399    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 14:55:21.827231 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.827209    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 14:55:21.893280 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.893228    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.893903 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.893889    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.893986 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.893920    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.893986 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.893936    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.893986 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.893962    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.901243 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.901229    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.901306 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.901248    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-157.ec2.internal\": node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:21.914595 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.914576    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:21.926206 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.926186    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"]
Sep 08 14:55:21.926270 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.926237    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.926870 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.926857    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.926954 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.926884    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.926954 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.926899    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.927173 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.927158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1ff1a9c0e7f9107b441cc843400901d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-157.ec2.internal\" (UID: \"e1ff1a9c0e7f9107b441cc843400901d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.929146 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929133    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.929265 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929239    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.929309 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929273    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.929772 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929760    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.929855 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929777    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.929855 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929781    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.929855 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929803    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.929855 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929817    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.929855 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.929824    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.931994 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.931978    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.932099 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.932000    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 14:55:21.932636 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.932621    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 14:55:21.932726 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.932651    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 14:55:21.932726 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:21.932660    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeHasSufficientPID"
Sep 08 14:55:21.951136 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.951121    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-157.ec2.internal\" not found" node="ip-10-0-136-157.ec2.internal"
Sep 08 14:55:21.955300 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:21.955285    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-157.ec2.internal\" not found" node="ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.014817 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.014800    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.027607 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.027590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1ff1a9c0e7f9107b441cc843400901d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-157.ec2.internal\" (UID: \"e1ff1a9c0e7f9107b441cc843400901d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.027653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.027622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e86e699226304aa7162930511476608-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal\" (UID: \"4e86e699226304aa7162930511476608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.027686 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.027658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e1ff1a9c0e7f9107b441cc843400901d-config\") pod \"kube-apiserver-proxy-ip-10-0-136-157.ec2.internal\" (UID: \"e1ff1a9c0e7f9107b441cc843400901d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.115237 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.115216    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.128452 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.128434    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e86e699226304aa7162930511476608-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal\" (UID: \"4e86e699226304aa7162930511476608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.128503 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.128460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e86e699226304aa7162930511476608-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal\" (UID: \"4e86e699226304aa7162930511476608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.128503 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.128488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4e86e699226304aa7162930511476608-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal\" (UID: \"4e86e699226304aa7162930511476608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.215876 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.215833    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.229140 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.229122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e86e699226304aa7162930511476608-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal\" (UID: \"4e86e699226304aa7162930511476608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.229183 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.229151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4e86e699226304aa7162930511476608-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal\" (UID: \"4e86e699226304aa7162930511476608\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.254280 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.254264    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.257614 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.257600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:22.316080 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.316058    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.416529 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.416511    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.517051 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.517012    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.617565 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.617544    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.664068 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.664050    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 14:55:22.664377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.664145    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 14:55:22.718511 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.718492    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.738595 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.738568    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 14:50:22 +0000 UTC" deadline="2028-03-26 17:52:05.217630031 +0000 UTC"
Sep 08 14:55:22.738595 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.738593    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13562h56m42.479040164s"
Sep 08 14:55:22.816404 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:22.816370    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ff1a9c0e7f9107b441cc843400901d.slice/crio-bfb5c47a67c8a51c0d20c1adac75e6e9368549fc30d874ae79cb5717b6d7f73a WatchSource:0}: Error finding container bfb5c47a67c8a51c0d20c1adac75e6e9368549fc30d874ae79cb5717b6d7f73a: Status 404 returned error can't find the container with id bfb5c47a67c8a51c0d20c1adac75e6e9368549fc30d874ae79cb5717b6d7f73a
Sep 08 14:55:22.818756 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.818741    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:22.820537 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.820508    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 14:55:22.828113 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:22.828081    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal" event={"ID":"e1ff1a9c0e7f9107b441cc843400901d","Type":"ContainerStarted","Data":"bfb5c47a67c8a51c0d20c1adac75e6e9368549fc30d874ae79cb5717b6d7f73a"}
Sep 08 14:55:22.919411 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:22.919392    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:23.019875 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.019857    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:23.093031 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.092985    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 14:55:23.098567 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.098551    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 14:55:23.120183 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.120163    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-157.ec2.internal\" not found"
Sep 08 14:55:23.137815 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.137439    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 14:55:23.226627 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.226608    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:23.234129 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.234110    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 14:55:23.235254 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.235243    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal"
Sep 08 14:55:23.243484 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.243472    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 14:55:23.707520 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.707368    2578 apiserver.go:52] "Watching apiserver"
Sep 08 14:55:23.714465 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.714444    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 14:55:23.714822 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.714804    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal","openshift-multus/multus-additional-cni-plugins-jtrfv","openshift-ovn-kubernetes/ovnkube-node-rbcfn","kube-system/konnectivity-agent-pll64","openshift-dns/node-resolver-dvvmx","openshift-multus/multus-7p728","openshift-multus/network-metrics-daemon-5kvpz","openshift-network-diagnostics/network-check-target-tsw5d","openshift-network-operator/iptables-alerter-shbd5","kube-system/global-pull-secret-syncer-pdmng","kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86","openshift-cluster-node-tuning-operator/tuned-2hwzr","openshift-image-registry/node-ca-z5wzq"]
Sep 08 14:55:23.718611 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.718594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:23.718672 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.718655    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:23.720755 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.720739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.723172 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.723154    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.723269 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.723213    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.723330 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.723279    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.723330 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.723312    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2rs6k\""
Sep 08 14:55:23.725521 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.725505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.726060 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.726043    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.726182 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.726167    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 14:55:23.727236 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 14:55:23.727236 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2s8xj\""
Sep 08 14:55:23.727236 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.727236 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727023    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 14:55:23.727236 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727074    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 14:55:23.727236 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727076    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 14:55:23.728130 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727790    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 14:55:23.728130 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.727998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.728130 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.728024    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cp9q7\""
Sep 08 14:55:23.728307 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.728261    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 14:55:23.730117 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.730097    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xp7gz\""
Sep 08 14:55:23.730211 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.730189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.730373 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.730356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.730670 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.730524    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.732592 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.732575    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rbfbn\""
Sep 08 14:55:23.732711 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.732681    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.732769 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.732724    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 14:55:23.732869 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.732852    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 14:55:23.732925 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.732908    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.733636 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.733618    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.735088 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.735073    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.735478 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.735456    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cjnkr\""
Sep 08 14:55:23.735631 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.735613    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.735926 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.735905    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.736222 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736204    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 14:55:23.736736 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736718    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-run-ovn-kubernetes\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.736785 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736760    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovnkube-script-lib\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.736830 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736796    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-cni-bin\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.736862 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:23.736911 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736897    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysconfig\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.736966 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysctl-d\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.736966 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736955    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysctl-conf\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737065 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.736984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-tuned\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737065 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-systemd-units\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737165 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737064    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-run-netns\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737165 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-env-overrides\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737165 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-run\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737165 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-sys\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-systemd\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovn-node-metrics-cert\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737238    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpxgc\" (UniqueName: \"kubernetes.io/projected/b17675ee-c49a-4ce0-810d-eef87a8edc93-kube-api-access-qpxgc\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737269    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb533fb6-ecee-4d0c-bb67-508a0b38e28e-konnectivity-ca\") pod \"konnectivity-agent-pll64\" (UID: \"cb533fb6-ecee-4d0c-bb67-508a0b38e28e\") " pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737302    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a24eeded-2925-4be8-b7f6-1ffa604cce09-hosts-file\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpz7\" (UniqueName: \"kubernetes.io/projected/a24eeded-2925-4be8-b7f6-1ffa604cce09-kube-api-access-5tpz7\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.737377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-ovn\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-tmp\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q626t\" (UniqueName: \"kubernetes.io/projected/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-kube-api-access-q626t\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-var-lib-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-slash\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737532    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-cni-netd\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb533fb6-ecee-4d0c-bb67-508a0b38e28e-agent-certs\") pod \"konnectivity-agent-pll64\" (UID: \"cb533fb6-ecee-4d0c-bb67-508a0b38e28e\") " pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737616    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-etc-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-systemd\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-node-log\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovnkube-config\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-modprobe-d\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.737783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-lib-modules\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737818    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-var-lib-kubelet\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-log-socket\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737913    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24eeded-2925-4be8-b7f6-1ffa604cce09-tmp-dir\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737943    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6k7\" (UniqueName: \"kubernetes.io/projected/6d9e2fd8-056e-49d3-a11b-70369b073da9-kube-api-access-tm6k7\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.737977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-kubernetes\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.738007    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-host\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.738394 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.738045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-kubelet\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.738964 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.738580    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.739646 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.739622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxfsf\""
Sep 08 14:55:23.739757 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.739721    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.740420 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.739914    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 14:55:23.740420 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.739948    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.740994 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.740974    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 14:55:23.741163 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.741125    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rr2gz\""
Sep 08 14:55:23.741240 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.741188    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 14:55:23.742149 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.742133    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.742251 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.742195    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:23.744668 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.744650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:23.744759 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.744717    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:23.745639 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.745619    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 14:55:23.747041 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.747021    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.749411 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.749381    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 14:55:23.749650 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.749629    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 14:55:23.749746 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.749678    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wv8vg\""
Sep 08 14:55:23.749885 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.749867    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 14:55:23.764380 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.764359    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjj82"
Sep 08 14:55:23.771169 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.771141    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qjj82"
Sep 08 14:55:23.828672 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.828650    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 14:55:23.839328 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-iptables-alerter-script\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.839449 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839352    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:23.839449 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-systemd\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.839592 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-node-log\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.839592 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-systemd\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.839592 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839528    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-cni-bin\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.839592 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-multus-certs\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-node-log\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgfr2\" (UniqueName: \"kubernetes.io/projected/1ce05732-a7c0-43e9-808e-58c906851b33-kube-api-access-pgfr2\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-var-lib-kubelet\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839705    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-kubelet\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.839788 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839769    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-run-ovn-kubernetes\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-var-lib-kubelet\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-kubelet\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.839841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovnkube-script-lib\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7935203-d452-4c42-8542-bdda6b8cd3f3-cni-binary-copy\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-run-ovn-kubernetes\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-sys-fs\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840135    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysconfig\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840170 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysctl-d\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysctl-conf\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysconfig\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840277    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-env-overrides\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysctl-conf\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovn-node-metrics-cert\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-sysctl-d\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-run\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb533fb6-ecee-4d0c-bb67-508a0b38e28e-konnectivity-ca\") pod \"konnectivity-agent-pll64\" (UID: \"cb533fb6-ecee-4d0c-bb67-508a0b38e28e\") " pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpz7\" (UniqueName: \"kubernetes.io/projected/a24eeded-2925-4be8-b7f6-1ffa604cce09-kube-api-access-5tpz7\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-socket-dir-parent\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840463    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-run\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-kubelet\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-tmp\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840577    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.840647 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-cni-netd\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-k8s-cni-cncf-io\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-etc-kubernetes\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovnkube-script-lib\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-cnibin\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-cni-netd\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-etc-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840812    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovnkube-config\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840869    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-etc-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.840951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cb533fb6-ecee-4d0c-bb67-508a0b38e28e-konnectivity-ca\") pod \"konnectivity-agent-pll64\" (UID: \"cb533fb6-ecee-4d0c-bb67-508a0b38e28e\") " pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841190    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841277    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-hostroot\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841315    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-daemon-config\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841331    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-env-overrides\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-registration-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.841531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htsv8\" (UniqueName: \"kubernetes.io/projected/31a43ec5-d33b-4b9f-af95-909e4fb379dd-kube-api-access-htsv8\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovnkube-config\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-cni-multus\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-modprobe-d\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-lib-modules\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841563    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-log-socket\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-modprobe-d\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-log-socket\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24eeded-2925-4be8-b7f6-1ffa604cce09-tmp-dir\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-cnibin\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841689    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-socket-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-lib-modules\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6k7\" (UniqueName: \"kubernetes.io/projected/6d9e2fd8-056e-49d3-a11b-70369b073da9-kube-api-access-tm6k7\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-kubernetes\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841797    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-host\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-netns\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-cni-bin\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.842351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-kubernetes\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-cni-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-host\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-host\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-serviceca\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68ff52bf-57a8-4714-a16b-b047539508d3-kubelet-config\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-cni-bin\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.841995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68ff52bf-57a8-4714-a16b-b047539508d3-dbus\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842007    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a24eeded-2925-4be8-b7f6-1ffa604cce09-tmp-dir\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt78q\" (UniqueName: \"kubernetes.io/projected/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-kube-api-access-nt78q\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842078    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-tuned\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-systemd-units\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842132    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-run-netns\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvjft\" (UniqueName: \"kubernetes.io/projected/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-kube-api-access-mvjft\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842173    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-systemd-units\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-sys\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.842208    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:23.843126 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842214    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-run-netns\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-systemd\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.842291    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:24.342258953 +0000 UTC m=+2.993683284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-sys\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-systemd\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpxgc\" (UniqueName: \"kubernetes.io/projected/b17675ee-c49a-4ce0-810d-eef87a8edc93-kube-api-access-qpxgc\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842497    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a24eeded-2925-4be8-b7f6-1ffa604cce09-hosts-file\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a24eeded-2925-4be8-b7f6-1ffa604cce09-hosts-file\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-system-cni-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg2j9\" (UniqueName: \"kubernetes.io/projected/f7935203-d452-4c42-8542-bdda6b8cd3f3-kube-api-access-cg2j9\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-host-slash\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-ovn\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q626t\" (UniqueName: \"kubernetes.io/projected/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-kube-api-access-q626t\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842737    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-var-lib-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-device-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-slash\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-var-lib-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.843960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.844863 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb533fb6-ecee-4d0c-bb67-508a0b38e28e-agent-certs\") pod \"konnectivity-agent-pll64\" (UID: \"cb533fb6-ecee-4d0c-bb67-508a0b38e28e\") " pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.844863 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-ovn\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.844863 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-conf-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.844863 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842926    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-host-slash\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.844863 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.842937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.844863 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.843060    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b17675ee-c49a-4ce0-810d-eef87a8edc93-run-openvswitch\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.845739 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.845486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-tmp\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.846800 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.845819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b17675ee-c49a-4ce0-810d-eef87a8edc93-ovn-node-metrics-cert\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.847364 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.847155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-etc-tuned\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.848585 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.848403    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cb533fb6-ecee-4d0c-bb67-508a0b38e28e-agent-certs\") pod \"konnectivity-agent-pll64\" (UID: \"cb533fb6-ecee-4d0c-bb67-508a0b38e28e\") " pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:23.849909 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.849886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpz7\" (UniqueName: \"kubernetes.io/projected/a24eeded-2925-4be8-b7f6-1ffa604cce09-kube-api-access-5tpz7\") pod \"node-resolver-dvvmx\" (UID: \"a24eeded-2925-4be8-b7f6-1ffa604cce09\") " pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:23.850897 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.850858    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6k7\" (UniqueName: \"kubernetes.io/projected/6d9e2fd8-056e-49d3-a11b-70369b073da9-kube-api-access-tm6k7\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:23.857514 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.855974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpxgc\" (UniqueName: \"kubernetes.io/projected/b17675ee-c49a-4ce0-810d-eef87a8edc93-kube-api-access-qpxgc\") pod \"ovnkube-node-rbcfn\" (UID: \"b17675ee-c49a-4ce0-810d-eef87a8edc93\") " pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:23.857514 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.856888    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q626t\" (UniqueName: \"kubernetes.io/projected/3052d9a5-1335-4e42-a980-c0bc44d7a7b3-kube-api-access-q626t\") pod \"tuned-2hwzr\" (UID: \"3052d9a5-1335-4e42-a980-c0bc44d7a7b3\") " pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:23.943450 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-cni-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.943450 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943409    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-host\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.943450 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-serviceca\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.943450 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68ff52bf-57a8-4714-a16b-b047539508d3-kubelet-config\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68ff52bf-57a8-4714-a16b-b047539508d3-dbus\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt78q\" (UniqueName: \"kubernetes.io/projected/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-kube-api-access-nt78q\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-host\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvjft\" (UniqueName: \"kubernetes.io/projected/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-kube-api-access-mvjft\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/68ff52bf-57a8-4714-a16b-b047539508d3-kubelet-config\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-system-cni-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943610    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg2j9\" (UniqueName: \"kubernetes.io/projected/f7935203-d452-4c42-8542-bdda6b8cd3f3-kube-api-access-cg2j9\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-host-slash\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943641    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/68ff52bf-57a8-4714-a16b-b047539508d3-dbus\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-device-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943686    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-host-slash\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-conf-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943735    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-iptables-alerter-script\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.943764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943754    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-cni-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-system-cni-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-device-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-conf-dir\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-cni-bin\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-cni-bin\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-multus-certs\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.943977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgfr2\" (UniqueName: \"kubernetes.io/projected/1ce05732-a7c0-43e9-808e-58c906851b33-kube-api-access-pgfr2\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944028    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944033    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-multus-certs\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944058    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7935203-d452-4c42-8542-bdda6b8cd3f3-cni-binary-copy\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-sys-fs\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-socket-dir-parent\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.944569 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-kubelet\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944245    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944270    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-iptables-alerter-script\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-k8s-cni-cncf-io\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944323    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-socket-dir-parent\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944327    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-etc-kubernetes\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-sys-fs\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944367    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-cnibin\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-k8s-cni-cncf-io\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-hostroot\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.944438    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-kubelet\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944449    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-daemon-config\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.945408 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-etc-kubernetes\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.944501    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:24.444484054 +0000 UTC m=+3.095908386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-system-cni-dir\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944585    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ce05732-a7c0-43e9-808e-58c906851b33-cnibin\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944629    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-serviceca\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944633    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-registration-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-registration-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htsv8\" (UniqueName: \"kubernetes.io/projected/31a43ec5-d33b-4b9f-af95-909e4fb379dd-kube-api-access-htsv8\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-cni-binary-copy\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7935203-d452-4c42-8542-bdda6b8cd3f3-cni-binary-copy\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-hostroot\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-etc-selinux\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-cni-multus\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-cnibin\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-socket-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944908    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-netns\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-cnibin\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944947    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f7935203-d452-4c42-8542-bdda6b8cd3f3-multus-daemon-config\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947961 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-var-lib-cni-multus\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947961 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.944965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f7935203-d452-4c42-8542-bdda6b8cd3f3-host-run-netns\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:23.947961 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.945015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ce05732-a7c0-43e9-808e-58c906851b33-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.947961 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.945014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/31a43ec5-d33b-4b9f-af95-909e4fb379dd-socket-dir\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.955180 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.955153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvjft\" (UniqueName: \"kubernetes.io/projected/4ce3e904-ba74-4c51-8f1f-e6d5de2129d6-kube-api-access-mvjft\") pod \"node-ca-z5wzq\" (UID: \"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6\") " pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:23.957623 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.956958    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:23.957623 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.956987    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:23.957623 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.957002    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:23.957623 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:23.957070    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:24.457053114 +0000 UTC m=+3.108477418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:23.959233 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.959208    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt78q\" (UniqueName: \"kubernetes.io/projected/8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c-kube-api-access-nt78q\") pod \"iptables-alerter-shbd5\" (UID: \"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c\") " pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:23.959731 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.959692    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htsv8\" (UniqueName: \"kubernetes.io/projected/31a43ec5-d33b-4b9f-af95-909e4fb379dd-kube-api-access-htsv8\") pod \"aws-ebs-csi-driver-node-dvv86\" (UID: \"31a43ec5-d33b-4b9f-af95-909e4fb379dd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:23.960244 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.960222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgfr2\" (UniqueName: \"kubernetes.io/projected/1ce05732-a7c0-43e9-808e-58c906851b33-kube-api-access-pgfr2\") pod \"multus-additional-cni-plugins-jtrfv\" (UID: \"1ce05732-a7c0-43e9-808e-58c906851b33\") " pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:23.960389 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:23.960370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg2j9\" (UniqueName: \"kubernetes.io/projected/f7935203-d452-4c42-8542-bdda6b8cd3f3-kube-api-access-cg2j9\") pod \"multus-7p728\" (UID: \"f7935203-d452-4c42-8542-bdda6b8cd3f3\") " pod="openshift-multus/multus-7p728"
Sep 08 14:55:24.030531 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.030503    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2hwzr"
Sep 08 14:55:24.041852 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.041653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:24.043392 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.043361    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3052d9a5_1335_4e42_a980_c0bc44d7a7b3.slice/crio-551967b7116ea8f515ad3701c7e1b9c7019a25ca9f8e87ad25b087ed28d1c9f1 WatchSource:0}: Error finding container 551967b7116ea8f515ad3701c7e1b9c7019a25ca9f8e87ad25b087ed28d1c9f1: Status 404 returned error can't find the container with id 551967b7116ea8f515ad3701c7e1b9c7019a25ca9f8e87ad25b087ed28d1c9f1
Sep 08 14:55:24.052686 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.052517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:24.056148 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.056125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dvvmx"
Sep 08 14:55:24.064937 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.064913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7p728"
Sep 08 14:55:24.065154 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.065136    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb533fb6_ecee_4d0c_bb67_508a0b38e28e.slice/crio-6da4fb4ac21c4ee7ab6a0c6f5903147a50e6cdd582cad78c478fdeffde8276e3 WatchSource:0}: Error finding container 6da4fb4ac21c4ee7ab6a0c6f5903147a50e6cdd582cad78c478fdeffde8276e3: Status 404 returned error can't find the container with id 6da4fb4ac21c4ee7ab6a0c6f5903147a50e6cdd582cad78c478fdeffde8276e3
Sep 08 14:55:24.066230 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.066202    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24eeded_2925_4be8_b7f6_1ffa604cce09.slice/crio-4debfa3b0ac02c6d3b68338fb09feadfad50246e031ee2a9ed190c69d598a726 WatchSource:0}: Error finding container 4debfa3b0ac02c6d3b68338fb09feadfad50246e031ee2a9ed190c69d598a726: Status 404 returned error can't find the container with id 4debfa3b0ac02c6d3b68338fb09feadfad50246e031ee2a9ed190c69d598a726
Sep 08 14:55:24.074264 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.074108    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shbd5"
Sep 08 14:55:24.074795 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.074766    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7935203_d452_4c42_8542_bdda6b8cd3f3.slice/crio-ef1a5677ea6074df9da485c00006da04c8b8c51b90abdbec41310eb8421111c5 WatchSource:0}: Error finding container ef1a5677ea6074df9da485c00006da04c8b8c51b90abdbec41310eb8421111c5: Status 404 returned error can't find the container with id ef1a5677ea6074df9da485c00006da04c8b8c51b90abdbec41310eb8421111c5
Sep 08 14:55:24.081421 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.081399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z5wzq"
Sep 08 14:55:24.084140 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.084117    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed90ed2_6f9c_423a_97d3_1c7aa7a7ec6c.slice/crio-7a29aa20d3311d7cea2ca41635484c8472df802e0a1cc5973a994420b955f5e3 WatchSource:0}: Error finding container 7a29aa20d3311d7cea2ca41635484c8472df802e0a1cc5973a994420b955f5e3: Status 404 returned error can't find the container with id 7a29aa20d3311d7cea2ca41635484c8472df802e0a1cc5973a994420b955f5e3
Sep 08 14:55:24.090514 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.090492    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce3e904_ba74_4c51_8f1f_e6d5de2129d6.slice/crio-67a77b2a4bb63a75edab30969bef617b3c216857fc492260743ccfb3846264da WatchSource:0}: Error finding container 67a77b2a4bb63a75edab30969bef617b3c216857fc492260743ccfb3846264da: Status 404 returned error can't find the container with id 67a77b2a4bb63a75edab30969bef617b3c216857fc492260743ccfb3846264da
Sep 08 14:55:24.091464 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.091316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jtrfv"
Sep 08 14:55:24.098381 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.098188    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86"
Sep 08 14:55:24.102665 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.102642    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce05732_a7c0_43e9_808e_58c906851b33.slice/crio-9e0f4fc87d249bd13e77091f9e9085513e844030fcb84871e04ac900b99bd069 WatchSource:0}: Error finding container 9e0f4fc87d249bd13e77091f9e9085513e844030fcb84871e04ac900b99bd069: Status 404 returned error can't find the container with id 9e0f4fc87d249bd13e77091f9e9085513e844030fcb84871e04ac900b99bd069
Sep 08 14:55:24.107895 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:24.107870    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a43ec5_d33b_4b9f_af95_909e4fb379dd.slice/crio-246a6b285c16cb5ff068b67b5c12d8a178ca72b2e33e8db6ae71e2865ae2a528 WatchSource:0}: Error finding container 246a6b285c16cb5ff068b67b5c12d8a178ca72b2e33e8db6ae71e2865ae2a528: Status 404 returned error can't find the container with id 246a6b285c16cb5ff068b67b5c12d8a178ca72b2e33e8db6ae71e2865ae2a528
Sep 08 14:55:24.248771 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.248693    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 14:55:24.348867 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.348826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:24.349048 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.348983    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:24.349108 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.349100    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:25.349079098 +0000 UTC m=+4.000503407 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:24.450103 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.450075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:24.450302 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.450223    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:24.450302 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.450300    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:25.450281131 +0000 UTC m=+4.101705439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:24.550947 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.550867    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:24.551125 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.551104    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:24.551176 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.551138    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:24.551176 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.551151    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:24.551262 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.551224    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:25.551203289 +0000 UTC m=+4.202627596 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:24.775073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.775016    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 14:50:24 +0000 UTC" deadline="2028-04-05 15:37:58.66415696 +0000 UTC"
Sep 08 14:55:24.775073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.775066    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13800h42m33.889095892s"
Sep 08 14:55:24.825754 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.825657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:24.825899 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:24.825790    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:24.835201 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.835160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pll64" event={"ID":"cb533fb6-ecee-4d0c-bb67-508a0b38e28e","Type":"ContainerStarted","Data":"6da4fb4ac21c4ee7ab6a0c6f5903147a50e6cdd582cad78c478fdeffde8276e3"}
Sep 08 14:55:24.838795 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.838771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"d97f771f370411f29e25169e183e9a212dff58ef7fbca489e6619a1d483b6bd1"}
Sep 08 14:55:24.842299 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.842218    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerStarted","Data":"9e0f4fc87d249bd13e77091f9e9085513e844030fcb84871e04ac900b99bd069"}
Sep 08 14:55:24.851503 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.847378    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5wzq" event={"ID":"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6","Type":"ContainerStarted","Data":"67a77b2a4bb63a75edab30969bef617b3c216857fc492260743ccfb3846264da"}
Sep 08 14:55:24.851503 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.850596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shbd5" event={"ID":"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c","Type":"ContainerStarted","Data":"7a29aa20d3311d7cea2ca41635484c8472df802e0a1cc5973a994420b955f5e3"}
Sep 08 14:55:24.852552 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.852396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7p728" event={"ID":"f7935203-d452-4c42-8542-bdda6b8cd3f3","Type":"ContainerStarted","Data":"ef1a5677ea6074df9da485c00006da04c8b8c51b90abdbec41310eb8421111c5"}
Sep 08 14:55:24.854235 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.854211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dvvmx" event={"ID":"a24eeded-2925-4be8-b7f6-1ffa604cce09","Type":"ContainerStarted","Data":"4debfa3b0ac02c6d3b68338fb09feadfad50246e031ee2a9ed190c69d598a726"}
Sep 08 14:55:24.855992 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.855968    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2hwzr" event={"ID":"3052d9a5-1335-4e42-a980-c0bc44d7a7b3","Type":"ContainerStarted","Data":"551967b7116ea8f515ad3701c7e1b9c7019a25ca9f8e87ad25b087ed28d1c9f1"}
Sep 08 14:55:24.858475 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:24.858453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86" event={"ID":"31a43ec5-d33b-4b9f-af95-909e4fb379dd","Type":"ContainerStarted","Data":"246a6b285c16cb5ff068b67b5c12d8a178ca72b2e33e8db6ae71e2865ae2a528"}
Sep 08 14:55:25.236995 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:25.236956    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e86e699226304aa7162930511476608.slice/crio-059a401a75a0f5393722134d518d12489d5e34fcaf6c12b40d488bc323af145f WatchSource:0}: Error finding container 059a401a75a0f5393722134d518d12489d5e34fcaf6c12b40d488bc323af145f: Status 404 returned error can't find the container with id 059a401a75a0f5393722134d518d12489d5e34fcaf6c12b40d488bc323af145f
Sep 08 14:55:25.358219 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.357992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:25.359025 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.358514    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:25.359025 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.358615    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:27.358594105 +0000 UTC m=+6.010018410 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:25.459131 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.458554    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:25.459131 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.458753    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:25.459131 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.458817    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:27.458797233 +0000 UTC m=+6.110221523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:25.559147 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.559108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:25.559324 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.559304    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:25.559395 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.559331    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:25.559395 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.559345    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:25.559495 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.559401    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:27.559384017 +0000 UTC m=+6.210808318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:25.775668 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.775523    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 14:50:24 +0000 UTC" deadline="2028-06-04 17:26:02.689208384 +0000 UTC"
Sep 08 14:55:25.775668 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.775604    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15242h30m36.913608564s"
Sep 08 14:55:25.828679 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.828648    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:25.828842 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.828786    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:25.829233 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.829210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:25.829357 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:25.829305    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:25.867695 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.867658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal" event={"ID":"e1ff1a9c0e7f9107b441cc843400901d","Type":"ContainerStarted","Data":"ab784683b1acd94e94d3bed9936357ce8d9527553fee0f7da0fde05749e02934"}
Sep 08 14:55:25.872524 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.872493    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal" event={"ID":"4e86e699226304aa7162930511476608","Type":"ContainerStarted","Data":"059a401a75a0f5393722134d518d12489d5e34fcaf6c12b40d488bc323af145f"}
Sep 08 14:55:25.883049 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:25.882994    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-157.ec2.internal" podStartSLOduration=2.882975384 podStartE2EDuration="2.882975384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:25.881856868 +0000 UTC m=+4.533281181" watchObservedRunningTime="2026-09-08 14:55:25.882975384 +0000 UTC m=+4.534399698"
Sep 08 14:55:26.825610 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:26.825578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:26.826048 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:26.825715    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:27.375301 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:27.374825    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:27.375301 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.374956    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:27.375301 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.375007    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:31.374993929 +0000 UTC m=+10.026418219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:27.476021 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:27.475983    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:27.476184 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.476121    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:27.476184 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.476163    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:31.476151731 +0000 UTC m=+10.127576030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:27.576594 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:27.576562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:27.576744 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.576721    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:27.576744 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.576741    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:27.576869 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.576750    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:27.576869 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.576799    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:31.57677996 +0000 UTC m=+10.228204257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:27.827193 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:27.826622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:27.827193 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.826765    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:27.827888 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:27.827725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:27.827888 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:27.827820    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:28.825536 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:28.825494    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:28.825773 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:28.825633    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:29.826699 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:29.826658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:29.827147 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:29.826804    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:29.827342 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:29.827322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:29.827440 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:29.827422    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:30.825836 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:30.825747    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:30.826002 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:30.825892    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:31.409765 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:31.409726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:31.410208 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.409906    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:31.410208 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.409984    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:39.40996398 +0000 UTC m=+18.061388279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:31.510211 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:31.510179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:31.510373 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.510328    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:31.510453 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.510389    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:39.51037141 +0000 UTC m=+18.161795713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:31.611646 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:31.611105    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:31.611646 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.611240    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:31.611646 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.611262    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:31.611646 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.611275    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:31.611646 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.611342    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:39.61132449 +0000 UTC m=+18.262748797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:31.826760 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:31.826618    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:31.827481 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.827442    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:31.827481 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:31.827467    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:31.827661 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:31.827578    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:32.826024 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:32.825941    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:32.826448 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:32.826090    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:33.826093 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:33.826064    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:33.826539 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:33.826197    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:33.826539 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:33.826262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:33.826539 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:33.826380    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:34.826400 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:34.826374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:34.826771 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:34.826486    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:35.825976 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:35.825943    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:35.826149 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:35.826072    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:35.826404 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:35.825942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:35.826491 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:35.826471    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:36.826438 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:36.826407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:36.826623 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:36.826539    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:37.826230 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:37.826195    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:37.826387 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:37.826337    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:37.826441 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:37.826384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:37.826559 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:37.826494    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:38.825598 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:38.825567    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:38.826083 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:38.825707    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:39.471380 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:39.471345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:39.471568 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.471516    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:39.471636 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.471602    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:55.471579386 +0000 UTC m=+34.123003683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:39.572637 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:39.572609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:39.572779 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.572760    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:39.572831 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.572816    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:55.572802804 +0000 UTC m=+34.224227094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:39.673445 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:39.673418    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:39.673615 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.673566    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:39.673615 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.673585    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:39.673615 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.673594    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:39.673742 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.673640    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:55.673625029 +0000 UTC m=+34.325049315 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:39.826243 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:39.826172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:39.826681 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.826305    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:39.826681 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:39.826363    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:39.826681 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:39.826465    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:40.826131 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:40.826102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:40.826294 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:40.826216    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:41.826665 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:41.826634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:41.827037 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:41.826754    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:41.827037 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:41.826808    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:41.827037 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:41.826918    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:42.827211 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.826497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:42.827211 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:42.826947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:42.915731 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.915696    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7p728" event={"ID":"f7935203-d452-4c42-8542-bdda6b8cd3f3","Type":"ContainerStarted","Data":"6a011d9bf0cb92b3f2ba1634d347cbbc4882b47f1fd18b0cdd6c01cf1361a388"}
Sep 08 14:55:42.917459 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.917431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2hwzr" event={"ID":"3052d9a5-1335-4e42-a980-c0bc44d7a7b3","Type":"ContainerStarted","Data":"9073fb41f839d9245b36652ad3477e889ced1a1936cd3cd86bfbefabaeb16115"}
Sep 08 14:55:42.920247 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.920220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"11a2b04c8ece3273d266f133c67aecf2467520e098e07f3f6540fb23fc63fc85"}
Sep 08 14:55:42.920333 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.920304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"fa30a8107f1ac06887f705011413760cf209e9d36ab347285bee9ab780d59682"}
Sep 08 14:55:42.920386 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.920357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"6abc7a05b42985801b7924975db36df87f39f9f59860a46c49b423b6742c6e67"}
Sep 08 14:55:42.920442 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.920404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"6756b28f0f8c62d266357170d8b1c6e57176e75d6aedcb9d493bb5186ba0d269"}
Sep 08 14:55:42.933823 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.933772    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7p728" podStartSLOduration=2.309248437 podStartE2EDuration="20.933756175s" totalImagesPullingTime="18.624507738s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:42.933319409 +0000 UTC m=+21.584743718" watchObservedRunningTime="2026-09-08 14:55:42.933756175 +0000 UTC m=+21.585180484"
Sep 08 14:55:42.951796 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:42.951731    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2hwzr" podStartSLOduration=2.532011256 podStartE2EDuration="20.951711408s" totalImagesPullingTime="18.419700152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:42.95139813 +0000 UTC m=+21.602822438" watchObservedRunningTime="2026-09-08 14:55:42.951711408 +0000 UTC m=+21.603135718"
Sep 08 14:55:43.825840 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.825810    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:43.825938 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:43.825907    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:43.826016 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.826000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:43.826094 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:43.826078    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:43.925339 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.925314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86" event={"ID":"31a43ec5-d33b-4b9f-af95-909e4fb379dd","Type":"ContainerStarted","Data":"d041125f50e48e187abce4b5e76c41b267b238811e562a0cabbaf861038fe238"}
Sep 08 14:55:43.926744 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.926723    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pll64" event={"ID":"cb533fb6-ecee-4d0c-bb67-508a0b38e28e","Type":"ContainerStarted","Data":"3829b48bf7e8a9a5aa0752112fb4238adc9f548d12db7a6f45bc1a2373567ac9"}
Sep 08 14:55:43.929799 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.929772    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"bf864828f8ec8666addaaf802bef2d4a9676de4e5ab4fd9da7e2c7e310c9df82"}
Sep 08 14:55:43.929901 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.929851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"e3af648c49d98c4ff8d8e749627451f70df922fa146dbaa17d67aca35759f64f"}
Sep 08 14:55:43.931141 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.931123    2578 generic.go:356] "Generic (PLEG): container finished" podID="4e86e699226304aa7162930511476608" containerID="22eb44210b060d6648174051092d74afe27f7cf475c29200dd643ed2ff80ddb9" exitCode=0
Sep 08 14:55:43.931230 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.931189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal" event={"ID":"4e86e699226304aa7162930511476608","Type":"ContainerDied","Data":"22eb44210b060d6648174051092d74afe27f7cf475c29200dd643ed2ff80ddb9"}
Sep 08 14:55:43.932732 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.932710    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ce05732-a7c0-43e9-808e-58c906851b33" containerID="fd238d88899e5bd8f1b82841601a13c3354dd698dfc54d4636d5644028d926a3" exitCode=0
Sep 08 14:55:43.932831 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.932787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerDied","Data":"fd238d88899e5bd8f1b82841601a13c3354dd698dfc54d4636d5644028d926a3"}
Sep 08 14:55:43.934707 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.934675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z5wzq" event={"ID":"4ce3e904-ba74-4c51-8f1f-e6d5de2129d6","Type":"ContainerStarted","Data":"09b3bfa790c29ce48d400b58707e46ad956125fe26c7e8c501d37042477f1d6a"}
Sep 08 14:55:43.936132 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.936107    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shbd5" event={"ID":"8ed90ed2-6f9c-423a-97d3-1c7aa7a7ec6c","Type":"ContainerStarted","Data":"36309b8bd34684c2462e58d7abe31746fdf550abbee5eb00fa88275e29a8410c"}
Sep 08 14:55:43.937650 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.937547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dvvmx" event={"ID":"a24eeded-2925-4be8-b7f6-1ffa604cce09","Type":"ContainerStarted","Data":"183c6123f6e4279bf6cfb27b3e51f4bbc73f63b02813ab85f653b81603fbb7ae"}
Sep 08 14:55:43.941318 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.941273    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pll64" podStartSLOduration=3.537408889 podStartE2EDuration="21.941254145s" totalImagesPullingTime="18.403845256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:43.940633135 +0000 UTC m=+22.592057445" watchObservedRunningTime="2026-09-08 14:55:43.941254145 +0000 UTC m=+22.592678453"
Sep 08 14:55:43.946918 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.946897    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 14:55:43.955809 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.955773    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z5wzq" podStartSLOduration=3.623292147 podStartE2EDuration="21.955761643s" totalImagesPullingTime="18.332469496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:43.955205283 +0000 UTC m=+22.606629591" watchObservedRunningTime="2026-09-08 14:55:43.955761643 +0000 UTC m=+22.607185986"
Sep 08 14:55:43.993558 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:43.993394    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dvvmx" podStartSLOduration=3.594438605 podStartE2EDuration="21.993383145s" totalImagesPullingTime="18.39894454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:43.993357238 +0000 UTC m=+22.644781546" watchObservedRunningTime="2026-09-08 14:55:43.993383145 +0000 UTC m=+22.644807452"
Sep 08 14:55:44.020883 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.020850    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-shbd5" podStartSLOduration=3.682712602 podStartE2EDuration="22.020839439s" totalImagesPullingTime="18.338126837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:44.020464415 +0000 UTC m=+22.671888723" watchObservedRunningTime="2026-09-08 14:55:44.020839439 +0000 UTC m=+22.672263746"
Sep 08 14:55:44.047461 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.047364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:44.115580 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.115556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:44.215971 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.215938    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:44.625873 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.625841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:44.808123 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.808027    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T14:55:43.946914672Z","UUID":"c41d08bd-bad4-4431-b7a9-a67f5f1ebde6","Handler":null,"Name":"","Endpoint":""}
Sep 08 14:55:44.809763 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.809742    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 14:55:44.809881 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.809774    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 14:55:44.825877 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.825845    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:44.825993 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:44.825971    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:44.942607 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.942355    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal" event={"ID":"4e86e699226304aa7162930511476608","Type":"ContainerStarted","Data":"8df44df70c5739dc8df07e3de193d31faf13361c93d5cac6b630c77371328867"}
Sep 08 14:55:44.944682 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.944656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86" event={"ID":"31a43ec5-d33b-4b9f-af95-909e4fb379dd","Type":"ContainerStarted","Data":"e23867a8f5dcbdf95ee68553b3bc7c6c8ccf0069183b3a241b9d30d88d0e0411"}
Sep 08 14:55:44.958149 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:44.958111    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-157.ec2.internal" podStartSLOduration=21.958097565 podStartE2EDuration="21.958097565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:44.957687749 +0000 UTC m=+23.609112059" watchObservedRunningTime="2026-09-08 14:55:44.958097565 +0000 UTC m=+23.609521873"
Sep 08 14:55:45.656547 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.656452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:45.657087 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.657065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:45.826577 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.826546    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:45.826726 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:45.826654    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:45.826786 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.826728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:45.826860 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:45.826842    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:45.949588 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.949505    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86" event={"ID":"31a43ec5-d33b-4b9f-af95-909e4fb379dd","Type":"ContainerStarted","Data":"f025ba45380575ddd50442fbbc41269142a8c6fb173e0b389d530563faee39cb"}
Sep 08 14:55:45.954157 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.954118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"b183e0cd4ce8a8329db851e9363b2f85c20de0442a097509553182b3e539982d"}
Sep 08 14:55:45.954303 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.954289    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:45.954753 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.954733    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pll64"
Sep 08 14:55:45.969310 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:45.969039    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dvv86" podStartSLOduration=3.607024004 podStartE2EDuration="23.96901696s" totalImagesPullingTime="20.361992956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:45.965749073 +0000 UTC m=+24.617173374" watchObservedRunningTime="2026-09-08 14:55:45.96901696 +0000 UTC m=+24.620441268"
Sep 08 14:55:46.790254 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:46.790225    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:46.825806 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:46.825778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:46.825944 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:46.825886    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:46.888702 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:46.888672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:47.825964 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.825801    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:47.825964 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:47.825918    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:47.825964 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.825800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:47.826944 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:47.826036    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:47.961745 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.961719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" event={"ID":"b17675ee-c49a-4ce0-810d-eef87a8edc93","Type":"ContainerStarted","Data":"9795e6716828235ddf84828a79e2ca759ab4a7340a8e99ac1fcedbf325f0351e"}
Sep 08 14:55:47.963466 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.963445    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:47.963680 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.963663    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:47.982616 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.982271    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:47.989857 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:47.989372    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn" podStartSLOduration=7.578422333 podStartE2EDuration="25.989353153s" totalImagesPullingTime="18.41093082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:55:47.988815409 +0000 UTC m=+26.640239719" watchObservedRunningTime="2026-09-08 14:55:47.989353153 +0000 UTC m=+26.640777462"
Sep 08 14:55:48.122703 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.122664    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:48.223051 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.223029    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:48.826031 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.826010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:48.826360 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:48.826092    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:48.967332 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.967293    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ce05732-a7c0-43e9-808e-58c906851b33" containerID="a6b53cacdf70b0d5eac1e374052bf077b20874149546fcb78e15abd8846922c4" exitCode=0
Sep 08 14:55:48.967429 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.967343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerDied","Data":"a6b53cacdf70b0d5eac1e374052bf077b20874149546fcb78e15abd8846922c4"}
Sep 08 14:55:48.967804 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.967786    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:48.981837 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:48.981821    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:55:49.712136 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:49.711640    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pdmng"]
Sep 08 14:55:49.712136 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:49.711751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:49.712136 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:49.711834    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:49.715755 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:49.715726    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kvpz"]
Sep 08 14:55:49.715889 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:49.715877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:49.715980 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:49.715964    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:49.716516 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:49.716493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tsw5d"]
Sep 08 14:55:49.716636 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:49.716605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:49.716708 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:49.716689    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:50.826387 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:50.826244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:50.826845 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:50.826458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:50.973710 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:50.973688    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ce05732-a7c0-43e9-808e-58c906851b33" containerID="1eb91149ae111f6e95746165f6cb5ff330b33e3d5ba6c249bd31a21f54880506" exitCode=0
Sep 08 14:55:50.973813 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:50.973758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerDied","Data":"1eb91149ae111f6e95746165f6cb5ff330b33e3d5ba6c249bd31a21f54880506"}
Sep 08 14:55:51.826460 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:51.826429    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:51.826855 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:51.826518    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:51.826855 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:51.826545    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:51.826855 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:51.826614    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:51.977978 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:51.977950    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ce05732-a7c0-43e9-808e-58c906851b33" containerID="bfd86cf11fb5bc4af3dc1aff93d2a214c477158844c16e59323b055684996909" exitCode=0
Sep 08 14:55:51.978073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:51.978025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerDied","Data":"bfd86cf11fb5bc4af3dc1aff93d2a214c477158844c16e59323b055684996909"}
Sep 08 14:55:52.787934 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:52.787894    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:52.826151 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:52.826131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:52.826294 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:52.826264    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:52.889441 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:52.889407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:53.825653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:53.825622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:53.825815 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:53.825632    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:53.825815 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:53.825740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pdmng" podUID="68ff52bf-57a8-4714-a16b-b047539508d3"
Sep 08 14:55:53.825931 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:53.825826    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tsw5d" podUID="980b82b5-e2fd-49c5-8c89-6d94d281e3e6"
Sep 08 14:55:54.826456 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:54.826427    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:54.827001 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:54.826569    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5kvpz" podUID="6d9e2fd8-056e-49d3-a11b-70369b073da9"
Sep 08 14:55:55.130577 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.130498    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:55.493778 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.493748    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:55.493926 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.493860    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:55.493926 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.493921    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs podName:6d9e2fd8-056e-49d3-a11b-70369b073da9 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:27.493902123 +0000 UTC m=+66.145326412 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs") pod "network-metrics-daemon-5kvpz" (UID: "6d9e2fd8-056e-49d3-a11b-70369b073da9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 14:55:55.567351 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.567324    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-157.ec2.internal" event="NodeReady"
Sep 08 14:55:55.567510 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.567439    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 14:55:55.594572 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.594548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:55.594716 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.594647    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:55.594716 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.594691    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret podName:68ff52bf-57a8-4714-a16b-b047539508d3 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:27.59467966 +0000 UTC m=+66.246103958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret") pod "global-pull-secret-syncer-pdmng" (UID: "68ff52bf-57a8-4714-a16b-b047539508d3") : object "kube-system"/"original-pull-secret" not registered
Sep 08 14:55:55.610601 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.610575    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zgn5w"]
Sep 08 14:55:55.636172 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.636103    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9rlmc"]
Sep 08 14:55:55.636402 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.636379    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.638441 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.638417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 14:55:55.638615 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.638596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 14:55:55.638691 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.638669    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hknbk\""
Sep 08 14:55:55.650330 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.650312    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgn5w"]
Sep 08 14:55:55.650423 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.650405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9rlmc"]
Sep 08 14:55:55.650484 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.650412    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.652617 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.652598    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 14:55:55.652724 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.652679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 14:55:55.652838 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.652823    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z5hlj\""
Sep 08 14:55:55.652910 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.652839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 14:55:55.695373 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.695347    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:55.695553 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.695535    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 14:55:55.695622 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.695563    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 14:55:55.695622 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.695577    2578 projected.go:196] Error preparing data for projected volume kube-api-access-9zfjb for pod openshift-network-diagnostics/network-check-target-tsw5d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:55.695748 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.695636    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb podName:980b82b5-e2fd-49c5-8c89-6d94d281e3e6 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:27.695618668 +0000 UTC m=+66.347042971 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zfjb" (UniqueName: "kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb") pod "network-check-target-tsw5d" (UID: "980b82b5-e2fd-49c5-8c89-6d94d281e3e6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 14:55:55.726204 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.726180    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:55.796677 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.796643    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc8d39ce-77a7-421e-8789-cd4f0173f426-cert\") pod \"ingress-canary-9rlmc\" (UID: \"bc8d39ce-77a7-421e-8789-cd4f0173f426\") " pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.796774 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.796683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84bj\" (UniqueName: \"kubernetes.io/projected/aff4ee34-7afb-49df-9b72-ef2e90a39866-kube-api-access-l84bj\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.796774 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.796707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4ee34-7afb-49df-9b72-ef2e90a39866-tmp-dir\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.796774 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.796731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8cbn\" (UniqueName: \"kubernetes.io/projected/bc8d39ce-77a7-421e-8789-cd4f0173f426-kube-api-access-x8cbn\") pod \"ingress-canary-9rlmc\" (UID: \"bc8d39ce-77a7-421e-8789-cd4f0173f426\") " pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.796774 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.796761    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4ee34-7afb-49df-9b72-ef2e90a39866-config-volume\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.796922 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.796779    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.826324 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.825971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:55:55.826324 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.826214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:55:55.828509 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.828489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 14:55:55.828874 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.828491    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s4zv9\""
Sep 08 14:55:55.828874 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.828553    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 14:55:55.828874 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.828576    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 14:55:55.897463 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.897410    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc8d39ce-77a7-421e-8789-cd4f0173f426-cert\") pod \"ingress-canary-9rlmc\" (UID: \"bc8d39ce-77a7-421e-8789-cd4f0173f426\") " pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.897463 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.897438    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84bj\" (UniqueName: \"kubernetes.io/projected/aff4ee34-7afb-49df-9b72-ef2e90a39866-kube-api-access-l84bj\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.897635 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.897463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4ee34-7afb-49df-9b72-ef2e90a39866-tmp-dir\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.897635 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.897492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8cbn\" (UniqueName: \"kubernetes.io/projected/bc8d39ce-77a7-421e-8789-cd4f0173f426-kube-api-access-x8cbn\") pod \"ingress-canary-9rlmc\" (UID: \"bc8d39ce-77a7-421e-8789-cd4f0173f426\") " pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.897635 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.897512    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4ee34-7afb-49df-9b72-ef2e90a39866-config-volume\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.897635 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.897551    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.897800 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.897658    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 14:55:55.897800 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:55.897709    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls podName:aff4ee34-7afb-49df-9b72-ef2e90a39866 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:56.397693671 +0000 UTC m=+35.049117957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls") pod "dns-default-zgn5w" (UID: "aff4ee34-7afb-49df-9b72-ef2e90a39866") : secret "dns-default-metrics-tls" not found
Sep 08 14:55:55.898409 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.898161    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aff4ee34-7afb-49df-9b72-ef2e90a39866-tmp-dir\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.898409 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.898367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aff4ee34-7afb-49df-9b72-ef2e90a39866-config-volume\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.902225 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.902207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc8d39ce-77a7-421e-8789-cd4f0173f426-cert\") pod \"ingress-canary-9rlmc\" (UID: \"bc8d39ce-77a7-421e-8789-cd4f0173f426\") " pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.905785 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.905769    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84bj\" (UniqueName: \"kubernetes.io/projected/aff4ee34-7afb-49df-9b72-ef2e90a39866-kube-api-access-l84bj\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:55.906033 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.906011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8cbn\" (UniqueName: \"kubernetes.io/projected/bc8d39ce-77a7-421e-8789-cd4f0173f426-kube-api-access-x8cbn\") pod \"ingress-canary-9rlmc\" (UID: \"bc8d39ce-77a7-421e-8789-cd4f0173f426\") " pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.961044 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.961016    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9rlmc"
Sep 08 14:55:55.993188 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:55.993162    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"]
Sep 08 14:55:56.011962 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.011943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"]
Sep 08 14:55:56.012075 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.012048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.014297 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.014271    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 08 14:55:56.014643 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.014623    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 08 14:55:56.014732 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.014658    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 08 14:55:56.014732 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.014683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 08 14:55:56.014955 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.014939    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-vf7hq\""
Sep 08 14:55:56.098611 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.098585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee890970-ffe2-4317-a669-1230e0317745-serving-cert\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.098730 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.098661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee890970-ffe2-4317-a669-1230e0317745-config\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.098730 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.098687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2k9\" (UniqueName: \"kubernetes.io/projected/ee890970-ffe2-4317-a669-1230e0317745-kube-api-access-8l2k9\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.114385 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.114361    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-556b5cd57d-klghc"]
Sep 08 14:55:56.140035 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.140014    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-556b5cd57d-klghc"]
Sep 08 14:55:56.140143 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.140111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.143402 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.143382    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 14:55:56.143849 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.143831    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 14:55:56.143938 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.143864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7z5lp\""
Sep 08 14:55:56.143938 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.143887    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 14:55:56.144168 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.144149    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 14:55:56.144256 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.144187    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 14:55:56.144314 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.144192    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 14:55:56.199560 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.199508    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee890970-ffe2-4317-a669-1230e0317745-serving-cert\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.199688 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.199578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee890970-ffe2-4317-a669-1230e0317745-config\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.199688 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.199606    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2k9\" (UniqueName: \"kubernetes.io/projected/ee890970-ffe2-4317-a669-1230e0317745-kube-api-access-8l2k9\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.200110 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.200087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ee890970-ffe2-4317-a669-1230e0317745-config\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.202574 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.202552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ee890970-ffe2-4317-a669-1230e0317745-serving-cert\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.211000 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.210979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2k9\" (UniqueName: \"kubernetes.io/projected/ee890970-ffe2-4317-a669-1230e0317745-kube-api-access-8l2k9\") pod \"service-ca-operator-859f94c4b-fnkl8\" (UID: \"ee890970-ffe2-4317-a669-1230e0317745\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.298884 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.298859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:56.300754 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.300714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-default-certificate\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.300754 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.300741    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-stats-auth\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.300899 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.300816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.300899 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.300859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.300990 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.300952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lcb\" (UniqueName: \"kubernetes.io/projected/593351b6-a625-467e-98e2-acbb80c3096d-kube-api-access-59lcb\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.321820 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.321803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"
Sep 08 14:55:56.399475 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.399450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:56.401557 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.401507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lcb\" (UniqueName: \"kubernetes.io/projected/593351b6-a625-467e-98e2-acbb80c3096d-kube-api-access-59lcb\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.401648 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.401598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-default-certificate\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.401648 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.401624    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-stats-auth\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.401736 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.401660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:56.401795 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.401761    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 14:55:56.401849 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.401803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.401849 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.401810    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls podName:aff4ee34-7afb-49df-9b72-ef2e90a39866 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:57.401795005 +0000 UTC m=+36.053219308 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls") pod "dns-default-zgn5w" (UID: "aff4ee34-7afb-49df-9b72-ef2e90a39866") : secret "dns-default-metrics-tls" not found
Sep 08 14:55:56.401849 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.401843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.401999 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.401938    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 14:55:56.401999 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.401945    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:55:56.901925101 +0000 UTC m=+35.553349410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : configmap references non-existent config key: service-ca.crt
Sep 08 14:55:56.401999 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.401991    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:55:56.901977758 +0000 UTC m=+35.553402048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : secret "router-metrics-certs-default" not found
Sep 08 14:55:56.404196 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.404175    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-default-certificate\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.404345 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.404326    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-stats-auth\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.409854 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.409837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lcb\" (UniqueName: \"kubernetes.io/projected/593351b6-a625-467e-98e2-acbb80c3096d-kube-api-access-59lcb\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.826232 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.826079    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:55:56.828739 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.828720    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 14:55:56.829065 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.828927    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4wbt\""
Sep 08 14:55:56.905799 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.905777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.905894 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.905804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:56.905944 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.905933    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:55:57.905917937 +0000 UTC m=+36.557342247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : configmap references non-existent config key: service-ca.crt
Sep 08 14:55:56.905997 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.905985    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 14:55:56.906050 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:56.906039    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:55:57.906028056 +0000 UTC m=+36.557452342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : secret "router-metrics-certs-default" not found
Sep 08 14:55:56.964783 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.964720    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:56.997177 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.996854    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9rlmc"]
Sep 08 14:55:56.997780 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:56.997760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8"]
Sep 08 14:55:57.001051 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:57.001028    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8d39ce_77a7_421e_8789_cd4f0173f426.slice/crio-78711faa459789fd774f2e4c41498a8bf5e524b70e81b86d8caa8cc549374173 WatchSource:0}: Error finding container 78711faa459789fd774f2e4c41498a8bf5e524b70e81b86d8caa8cc549374173: Status 404 returned error can't find the container with id 78711faa459789fd774f2e4c41498a8bf5e524b70e81b86d8caa8cc549374173
Sep 08 14:55:57.001737 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:55:57.001645    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee890970_ffe2_4317_a669_1230e0317745.slice/crio-7293e4e98fda41923135d7a6ac33eac3fb2e781911400af610e92550f05379eb WatchSource:0}: Error finding container 7293e4e98fda41923135d7a6ac33eac3fb2e781911400af610e92550f05379eb: Status 404 returned error can't find the container with id 7293e4e98fda41923135d7a6ac33eac3fb2e781911400af610e92550f05379eb
Sep 08 14:55:57.066215 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.066190    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:57.410181 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.410156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:57.410318 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:57.410302    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 14:55:57.410373 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:57.410364    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls podName:aff4ee34-7afb-49df-9b72-ef2e90a39866 nodeName:}" failed. No retries permitted until 2026-09-08 14:55:59.410348077 +0000 UTC m=+38.061772366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls") pod "dns-default-zgn5w" (UID: "aff4ee34-7afb-49df-9b72-ef2e90a39866") : secret "dns-default-metrics-tls" not found
Sep 08 14:55:57.914457 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.914422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:57.914457 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.914462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:57.915116 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:57.914604    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 14:55:57.915116 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:57.914628    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:55:59.91460248 +0000 UTC m=+38.566026789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : configmap references non-existent config key: service-ca.crt
Sep 08 14:55:57.915116 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:57.914655    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:55:59.914645316 +0000 UTC m=+38.566069605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : secret "router-metrics-certs-default" not found
Sep 08 14:55:57.992463 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.992435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9rlmc" event={"ID":"bc8d39ce-77a7-421e-8789-cd4f0173f426","Type":"ContainerStarted","Data":"78711faa459789fd774f2e4c41498a8bf5e524b70e81b86d8caa8cc549374173"}
Sep 08 14:55:57.995861 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.995832    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ce05732-a7c0-43e9-808e-58c906851b33" containerID="87d54a710ecccefe4ccd943cd17d090760820dff36cf5dab01d7ae831ed31d99" exitCode=0
Sep 08 14:55:57.995988 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.995919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerDied","Data":"87d54a710ecccefe4ccd943cd17d090760820dff36cf5dab01d7ae831ed31d99"}
Sep 08 14:55:57.997458 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:57.997438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8" event={"ID":"ee890970-ffe2-4317-a669-1230e0317745","Type":"ContainerStarted","Data":"7293e4e98fda41923135d7a6ac33eac3fb2e781911400af610e92550f05379eb"}
Sep 08 14:55:59.004473 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.004432    2578 generic.go:356] "Generic (PLEG): container finished" podID="1ce05732-a7c0-43e9-808e-58c906851b33" containerID="1d37fba3587fb531e2c71ec42dbfcc1a4b176cb877ea9f4da8deb5f9b889675a" exitCode=0
Sep 08 14:55:59.005203 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.004507    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerDied","Data":"1d37fba3587fb531e2c71ec42dbfcc1a4b176cb877ea9f4da8deb5f9b889675a"}
Sep 08 14:55:59.426930 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.426755    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:55:59.427049 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:59.426902    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 14:55:59.427097 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:59.427077    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls podName:aff4ee34-7afb-49df-9b72-ef2e90a39866 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:03.427054103 +0000 UTC m=+42.078478403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls") pod "dns-default-zgn5w" (UID: "aff4ee34-7afb-49df-9b72-ef2e90a39866") : secret "dns-default-metrics-tls" not found
Sep 08 14:55:59.706068 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.705960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:59.807287 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.807260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:55:59.930288 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.930256    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:59.930428 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:55:59.930301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:55:59.930497 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:59.930431    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:56:03.930408988 +0000 UTC m=+42.581833284 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : configmap references non-existent config key: service-ca.crt
Sep 08 14:55:59.930497 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:59.930452    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 14:55:59.930497 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:55:59.930497    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:56:03.930482758 +0000 UTC m=+42.581907045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : secret "router-metrics-certs-default" not found
Sep 08 14:56:00.009037 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.009002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9rlmc" event={"ID":"bc8d39ce-77a7-421e-8789-cd4f0173f426","Type":"ContainerStarted","Data":"7f90c60d97d3640dc09f752574005a741a7917b5c6e9569901a65fd9aaf9cb26"}
Sep 08 14:56:00.015145 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.015121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" event={"ID":"1ce05732-a7c0-43e9-808e-58c906851b33","Type":"ContainerStarted","Data":"efc01b8537a8daecca15ec59ad44ac950dfffd337e673e8a5f01ff5a57b3c813"}
Sep 08 14:56:00.025278 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.025238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9rlmc" podStartSLOduration=2.711715995 podStartE2EDuration="5.025224983s" totalImagesPullingTime="2.313508988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:00.024451807 +0000 UTC m=+38.675876117" watchObservedRunningTime="2026-09-08 14:56:00.025224983 +0000 UTC m=+38.676649290"
Sep 08 14:56:00.045653 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.045606    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jtrfv" podStartSLOduration=13.536162933 podStartE2EDuration="38.045589331s" totalImagesPullingTime="24.509426398s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:00.044425446 +0000 UTC m=+38.695849758" watchObservedRunningTime="2026-09-08 14:56:00.045589331 +0000 UTC m=+38.697013637"
Sep 08 14:56:00.164362 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.164317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:00.264784 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.264719    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:00.761511 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.761468    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:00.862159 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:00.862131    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:01.540478 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.540450    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"]
Sep 08 14:56:01.542220 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.542183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:01.552518 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.552495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"
Sep 08 14:56:01.554222 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.554201    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"]
Sep 08 14:56:01.555861 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.555833    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-qss6f\""
Sep 08 14:56:01.555961 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.555943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 08 14:56:01.556633 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.556603    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 08 14:56:01.599485 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.599461    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bq2g2"]
Sep 08 14:56:01.619539 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.619505    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bq2g2"]
Sep 08 14:56:01.619622 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.619605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.621906 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.621890    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 14:56:01.622143 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.622073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 14:56:01.622143 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.622089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 14:56:01.622281 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.622089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 14:56:01.622281 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.622194    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xf5qg\""
Sep 08 14:56:01.626474 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.626454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:01.743093 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.743076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-crio-socket\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.743166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.743103    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-data-volume\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.743166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.743120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45w8\" (UniqueName: \"kubernetes.io/projected/db484c3f-cbab-423d-9ad4-d43d17f5ac49-kube-api-access-m45w8\") pod \"migrator-fdbb8fdff-nkfdj\" (UID: \"db484c3f-cbab-423d-9ad4-d43d17f5ac49\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"
Sep 08 14:56:01.743166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.743136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfr9r\" (UniqueName: \"kubernetes.io/projected/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-api-access-cfr9r\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.743271 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.743240    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.743314 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.743301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844237 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844370 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-crio-socket\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844370 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-data-volume\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844370 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45w8\" (UniqueName: \"kubernetes.io/projected/db484c3f-cbab-423d-9ad4-d43d17f5ac49-kube-api-access-m45w8\") pod \"migrator-fdbb8fdff-nkfdj\" (UID: \"db484c3f-cbab-423d-9ad4-d43d17f5ac49\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"
Sep 08 14:56:01.844370 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844330    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfr9r\" (UniqueName: \"kubernetes.io/projected/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-api-access-cfr9r\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844598 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844379    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844598 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:01.844389    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:01.844598 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:01.844451    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:02.344430688 +0000 UTC m=+40.995854988 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:01.844598 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-crio-socket\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.844598 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:01.844506    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 14:56:01.844598 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:01.844577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:02.344561089 +0000 UTC m=+40.995985392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : configmap "kube-rbac-proxy" not found
Sep 08 14:56:01.844824 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.844665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-data-volume\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.846364 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.846341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:01.853740 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.853723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfr9r\" (UniqueName: \"kubernetes.io/projected/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-api-access-cfr9r\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:01.853808 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.853775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45w8\" (UniqueName: \"kubernetes.io/projected/db484c3f-cbab-423d-9ad4-d43d17f5ac49-kube-api-access-m45w8\") pod \"migrator-fdbb8fdff-nkfdj\" (UID: \"db484c3f-cbab-423d-9ad4-d43d17f5ac49\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"
Sep 08 14:56:01.875583 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:01.875560    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"
Sep 08 14:56:02.021575 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:02.021547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8" event={"ID":"ee890970-ffe2-4317-a669-1230e0317745","Type":"ContainerStarted","Data":"17e96c270bd17639d9b86ac2aab44e8d77a0884958877000fd0b77ef6ccbcdc6"}
Sep 08 14:56:02.038279 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:02.038110    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8" podStartSLOduration=2.333270592 podStartE2EDuration="7.038093905s" totalImagesPullingTime="4.704823313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:02.03791777 +0000 UTC m=+40.689342077" watchObservedRunningTime="2026-09-08 14:56:02.038093905 +0000 UTC m=+40.689518207"
Sep 08 14:56:02.167088 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:02.167021    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj"]
Sep 08 14:56:02.169314 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:02.169292    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb484c3f_cbab_423d_9ad4_d43d17f5ac49.slice/crio-b43432c7f26289fa1ddcf6a9e7231afa66a5a9c5a0c18a3eefab05ce9fb76903 WatchSource:0}: Error finding container b43432c7f26289fa1ddcf6a9e7231afa66a5a9c5a0c18a3eefab05ce9fb76903: Status 404 returned error can't find the container with id b43432c7f26289fa1ddcf6a9e7231afa66a5a9c5a0c18a3eefab05ce9fb76903
Sep 08 14:56:02.349235 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:02.349209    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:02.349325 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:02.349267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:02.349380 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:02.349319    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 14:56:02.349380 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:02.349379    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:03.34936515 +0000 UTC m=+42.000789449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : configmap "kube-rbac-proxy" not found
Sep 08 14:56:02.349462 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:02.349393    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:02.349462 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:02.349438    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:03.349422115 +0000 UTC m=+42.000846405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:03.025089 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.025069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj" event={"ID":"db484c3f-cbab-423d-9ad4-d43d17f5ac49","Type":"ContainerStarted","Data":"b43432c7f26289fa1ddcf6a9e7231afa66a5a9c5a0c18a3eefab05ce9fb76903"}
Sep 08 14:56:03.343193 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.343056    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dvvmx_a24eeded-2925-4be8-b7f6-1ffa604cce09/dns-node-resolver/0.log"
Sep 08 14:56:03.356764 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.356724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:03.356910 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.356820    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 14:56:03.356910 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.356820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:03.356910 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.356877    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:05.356860332 +0000 UTC m=+44.008284631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : configmap "kube-rbac-proxy" not found
Sep 08 14:56:03.356910 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.356901    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:03.357084 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.356949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:05.356934142 +0000 UTC m=+44.008358429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:03.400329 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.400290    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:03.457746 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.457725    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:03.457885 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.457870    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 14:56:03.457952 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.457930    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls podName:aff4ee34-7afb-49df-9b72-ef2e90a39866 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:11.457914214 +0000 UTC m=+50.109338505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls") pod "dns-default-zgn5w" (UID: "aff4ee34-7afb-49df-9b72-ef2e90a39866") : secret "dns-default-metrics-tls" not found
Sep 08 14:56:03.459483 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.459463    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:03.650130 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.650052    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:03.749333 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.749300    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:03.850004 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.849979    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:03.961423 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.961377    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:03.961423 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:03.961400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:03.961548 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.961516    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:56:11.961501637 +0000 UTC m=+50.612925950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : configmap references non-existent config key: service-ca.crt
Sep 08 14:56:03.961595 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.961560    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 14:56:03.961638 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:03.961607    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:56:11.961594764 +0000 UTC m=+50.613019068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : secret "router-metrics-certs-default" not found
Sep 08 14:56:04.067303 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:04.067284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:04.167880 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:04.167855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:04.741623 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:04.741603    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5wzq_4ce3e904-ba74-4c51-8f1f-e6d5de2129d6/node-ca/0.log"
Sep 08 14:56:05.030680 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.030653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj" event={"ID":"db484c3f-cbab-423d-9ad4-d43d17f5ac49","Type":"ContainerStarted","Data":"3a1173dda4b886f975759b387b5fa178dbada024b39dc9867b29751aa7bbff4a"}
Sep 08 14:56:05.030760 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.030718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj" event={"ID":"db484c3f-cbab-423d-9ad4-d43d17f5ac49","Type":"ContainerStarted","Data":"c8ca6f031742a32de65717e227cfb91d77d8afaf3b3f822b22c0b5c23278f5a5"}
Sep 08 14:56:05.045413 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.045297    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-nkfdj" podStartSLOduration=2.157973062 podStartE2EDuration="4.045285308s" totalImagesPullingTime="1.887312246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:05.044509597 +0000 UTC m=+43.695933905" watchObservedRunningTime="2026-09-08 14:56:05.045285308 +0000 UTC m=+43.696709616"
Sep 08 14:56:05.125028 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.125001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:05.141372 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.141355    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9rlmc_bc8d39ce-77a7-421e-8789-cd4f0173f426/serve-healthcheck-canary/0.log"
Sep 08 14:56:05.184096 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.184074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:05.370374 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.370325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:05.370453 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.370386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:05.370453 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:05.370431    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 14:56:05.370571 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:05.370475    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:09.370462764 +0000 UTC m=+48.021887072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : configmap "kube-rbac-proxy" not found
Sep 08 14:56:05.370571 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:05.370481    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:05.370571 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:05.370517    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:09.370504801 +0000 UTC m=+48.021929088 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:05.688667 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.688620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:05.829727 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:05.829700    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:08.451804 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:08.451776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:09.395809 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:09.395787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:09.395942 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:09.395839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:09.395942 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:09.395895    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 14:56:09.395942 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:09.395932    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:09.396047 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:09.395948    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:17.395935207 +0000 UTC m=+56.047359493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : configmap "kube-rbac-proxy" not found
Sep 08 14:56:09.396047 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:09.395968    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:17.395958359 +0000 UTC m=+56.047382645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:10.050073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:10.050049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:10.178278 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:10.178253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:10.294370 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:10.294352    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:10.394906 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:10.394860    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:10.454502 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:10.454485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:11.511807 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:11.511787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:11.512106 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:11.511885    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 14:56:11.512106 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:11.511928    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls podName:aff4ee34-7afb-49df-9b72-ef2e90a39866 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:27.51191621 +0000 UTC m=+66.163340497 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls") pod "dns-default-zgn5w" (UID: "aff4ee34-7afb-49df-9b72-ef2e90a39866") : secret "dns-default-metrics-tls" not found
Sep 08 14:56:12.015554 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:12.015536    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:12.015650 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:12.015637    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:12.015690 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:12.015666    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 14:56:12.015734 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:12.015725    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:56:28.015698401 +0000 UTC m=+66.667122688 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : secret "router-metrics-certs-default" not found
Sep 08 14:56:12.015782 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:12.015740    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle podName:593351b6-a625-467e-98e2-acbb80c3096d nodeName:}" failed. No retries permitted until 2026-09-08 14:56:28.015734078 +0000 UTC m=+66.667158364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle") pod "router-default-556b5cd57d-klghc" (UID: "593351b6-a625-467e-98e2-acbb80c3096d") : configmap references non-existent config key: service-ca.crt
Sep 08 14:56:12.375447 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:12.375396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:14.119837 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:14.119780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:14.178577 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:14.178559    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:14.339806 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:14.339779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:14.541044 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:14.541015    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:16.180608 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:16.180579    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:16.235558 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:16.235516    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:16.322970 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:16.322951    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:16.378180 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:16.378159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:16.538042 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:16.538020    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:16.738956 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:16.738927    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:17.454779 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:17.454757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:17.455114 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:17.454816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:17.455114 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:17.454904    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:17.455114 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:17.454964    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls podName:eb0f489a-6369-42fb-8518-f1ac77cfbcb5 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:33.454949423 +0000 UTC m=+72.106373709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-bq2g2" (UID: "eb0f489a-6369-42fb-8518-f1ac77cfbcb5") : secret "insights-runtime-extractor-tls" not found
Sep 08 14:56:17.455303 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:17.455288    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:17.545337 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:17.545311    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:17.741907 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:17.741851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:18.139466 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:18.139444    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:18.210681 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:18.210660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:18.743343 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:18.743321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:19.142336 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:19.142315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:19.566136 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:19.566113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:19.945588 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:19.945539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:20.138166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:20.138147    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:20.537776 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:20.537740    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:20.735264 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:20.735246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:20.983934 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:20.983897    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rbcfn"
Sep 08 14:56:22.286874 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:22.286843    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:22.388277 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:22.388251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:22.488373 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:22.488350    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:22.601105 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:22.601050    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:22.966927 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:22.966862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:23.018777 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:23.018757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:24.666274 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:24.666247    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:24.729198 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:24.729175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:24.998591 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:24.998545    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:26.964959 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:26.964933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:27.047697 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.047674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:27.173981 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.173945    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"]
Sep 08 14:56:27.178273 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.178257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.180590 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.180567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 14:56:27.180916 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.180901    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 08 14:56:27.181014 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.180956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 08 14:56:27.181076 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.181028    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 08 14:56:27.181849 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.181835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 14:56:27.181964 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.181936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 14:56:27.182210 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.182199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 08 14:56:27.189402 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.189382    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"]
Sep 08 14:56:27.223446 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.223398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-hub\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.223446 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.223441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wxl\" (UniqueName: \"kubernetes.io/projected/15d20d48-1773-4f82-b4cf-9463167f3e43-kube-api-access-p2wxl\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.223613 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.223476    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d20d48-1773-4f82-b4cf-9463167f3e43-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.223613 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.223507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.223613 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.223539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.223731 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.223612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-ca\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.280246 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.280226    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:27.324219 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wxl\" (UniqueName: \"kubernetes.io/projected/15d20d48-1773-4f82-b4cf-9463167f3e43-kube-api-access-p2wxl\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.324324 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d20d48-1773-4f82-b4cf-9463167f3e43-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.324324 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.324435 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.324509 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-ca\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.324617 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-hub\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.324946 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.324917    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/15d20d48-1773-4f82-b4cf-9463167f3e43-additional-service-ca\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.326948 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.326922    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.327146 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.327127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-ca\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.327195 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.327148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.327195 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.327127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/15d20d48-1773-4f82-b4cf-9463167f3e43-hub\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.332577 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.332560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wxl\" (UniqueName: \"kubernetes.io/projected/15d20d48-1773-4f82-b4cf-9463167f3e43-kube-api-access-p2wxl\") pod \"cluster-proxy-proxy-agent-78997fc645-j9vvt\" (UID: \"15d20d48-1773-4f82-b4cf-9463167f3e43\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.369993 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.369964    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:27.494975 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.494918    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"
Sep 08 14:56:27.525684 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.525663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:27.525785 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.525736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:56:27.527701 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.527681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aff4ee34-7afb-49df-9b72-ef2e90a39866-metrics-tls\") pod \"dns-default-zgn5w\" (UID: \"aff4ee34-7afb-49df-9b72-ef2e90a39866\") " pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:27.527991 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.527976    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 14:56:27.539677 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.539628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d9e2fd8-056e-49d3-a11b-70369b073da9-metrics-certs\") pod \"network-metrics-daemon-5kvpz\" (UID: \"6d9e2fd8-056e-49d3-a11b-70369b073da9\") " pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:56:27.626149 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.626120    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:56:27.628374 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.628357    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 14:56:27.639189 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.639165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/68ff52bf-57a8-4714-a16b-b047539508d3-original-pull-secret\") pod \"global-pull-secret-syncer-pdmng\" (UID: \"68ff52bf-57a8-4714-a16b-b047539508d3\") " pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:56:27.645109 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.645095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pdmng"
Sep 08 14:56:27.727221 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.727196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:56:27.729474 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.729456    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 14:56:27.737248 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.737231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-j4wbt\""
Sep 08 14:56:27.739285 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.739272    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 14:56:27.746096 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.746054    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5kvpz"
Sep 08 14:56:27.749188 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.749142    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hknbk\""
Sep 08 14:56:27.750513 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.750494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zfjb\" (UniqueName: \"kubernetes.io/projected/980b82b5-e2fd-49c5-8c89-6d94d281e3e6-kube-api-access-9zfjb\") pod \"network-check-target-tsw5d\" (UID: \"980b82b5-e2fd-49c5-8c89-6d94d281e3e6\") " pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:56:27.757902 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.757880    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:27.761135 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.761116    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt"]
Sep 08 14:56:27.765010 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:27.764989    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15d20d48_1773_4f82_b4cf_9463167f3e43.slice/crio-e0b5377c1b475f166f488c12813348dd5d76c9da492a532128c0793b4a549a2c WatchSource:0}: Error finding container e0b5377c1b475f166f488c12813348dd5d76c9da492a532128c0793b4a549a2c: Status 404 returned error can't find the container with id e0b5377c1b475f166f488c12813348dd5d76c9da492a532128c0793b4a549a2c
Sep 08 14:56:27.910290 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.910267    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pdmng"]
Sep 08 14:56:27.913999 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:27.913966    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ff52bf_57a8_4714_a16b_b047539508d3.slice/crio-64412c6dc32b661988ff54cbf0e8c593fb8677c1a2b675e29cb01c6420cce635 WatchSource:0}: Error finding container 64412c6dc32b661988ff54cbf0e8c593fb8677c1a2b675e29cb01c6420cce635: Status 404 returned error can't find the container with id 64412c6dc32b661988ff54cbf0e8c593fb8677c1a2b675e29cb01c6420cce635
Sep 08 14:56:27.940402 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.940384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-s4zv9\""
Sep 08 14:56:27.948299 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:27.948284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:56:28.016295 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.016272    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5kvpz"]
Sep 08 14:56:28.028999 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.028976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:28.029105 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.029006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:28.029565 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.029517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/593351b6-a625-467e-98e2-acbb80c3096d-service-ca-bundle\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:28.031321 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.031303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/593351b6-a625-467e-98e2-acbb80c3096d-metrics-certs\") pod \"router-default-556b5cd57d-klghc\" (UID: \"593351b6-a625-467e-98e2-acbb80c3096d\") " pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:28.035851 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.035832    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zgn5w"]
Sep 08 14:56:28.038541 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:28.038503    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff4ee34_7afb_49df_9b72_ef2e90a39866.slice/crio-435e707506c5b6ab00005394aebfa1c9c8858c27dcd026e91d034627ee38c161 WatchSource:0}: Error finding container 435e707506c5b6ab00005394aebfa1c9c8858c27dcd026e91d034627ee38c161: Status 404 returned error can't find the container with id 435e707506c5b6ab00005394aebfa1c9c8858c27dcd026e91d034627ee38c161
Sep 08 14:56:28.081902 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.081880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kvpz" event={"ID":"6d9e2fd8-056e-49d3-a11b-70369b073da9","Type":"ContainerStarted","Data":"7c3ebf314d475b3786a30e22bb89586767691525376737b2402edfa52740eb91"}
Sep 08 14:56:28.082901 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.082885    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgn5w" event={"ID":"aff4ee34-7afb-49df-9b72-ef2e90a39866","Type":"ContainerStarted","Data":"435e707506c5b6ab00005394aebfa1c9c8858c27dcd026e91d034627ee38c161"}
Sep 08 14:56:28.085155 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.085132    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pdmng" event={"ID":"68ff52bf-57a8-4714-a16b-b047539508d3","Type":"ContainerStarted","Data":"64412c6dc32b661988ff54cbf0e8c593fb8677c1a2b675e29cb01c6420cce635"}
Sep 08 14:56:28.086725 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.086702    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt" event={"ID":"15d20d48-1773-4f82-b4cf-9463167f3e43","Type":"ContainerStarted","Data":"e0b5377c1b475f166f488c12813348dd5d76c9da492a532128c0793b4a549a2c"}
Sep 08 14:56:28.214317 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.214299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tsw5d"]
Sep 08 14:56:28.217397 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:28.217376    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980b82b5_e2fd_49c5_8c89_6d94d281e3e6.slice/crio-a2acc945196890b495c367bc7acaf5afc5214573e95007cf842c2da728ff3979 WatchSource:0}: Error finding container a2acc945196890b495c367bc7acaf5afc5214573e95007cf842c2da728ff3979: Status 404 returned error can't find the container with id a2acc945196890b495c367bc7acaf5afc5214573e95007cf842c2da728ff3979
Sep 08 14:56:28.251718 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.251665    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-7z5lp\""
Sep 08 14:56:28.260576 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.260556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:28.531482 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:28.531446    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-556b5cd57d-klghc"]
Sep 08 14:56:28.535256 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:28.535229    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod593351b6_a625_467e_98e2_acbb80c3096d.slice/crio-395c6c1782395be19cf720cc1d53bdd1df52cf57509fcc1c04ac68e6cee62cf9 WatchSource:0}: Error finding container 395c6c1782395be19cf720cc1d53bdd1df52cf57509fcc1c04ac68e6cee62cf9: Status 404 returned error can't find the container with id 395c6c1782395be19cf720cc1d53bdd1df52cf57509fcc1c04ac68e6cee62cf9
Sep 08 14:56:29.094098 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:29.093313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-556b5cd57d-klghc" event={"ID":"593351b6-a625-467e-98e2-acbb80c3096d","Type":"ContainerStarted","Data":"2ef8c69702928a7f1f2f71638db57edc214c4cd1741f7affa09e948979d31a10"}
Sep 08 14:56:29.094098 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:29.093368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-556b5cd57d-klghc" event={"ID":"593351b6-a625-467e-98e2-acbb80c3096d","Type":"ContainerStarted","Data":"395c6c1782395be19cf720cc1d53bdd1df52cf57509fcc1c04ac68e6cee62cf9"}
Sep 08 14:56:29.096857 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:29.096813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tsw5d" event={"ID":"980b82b5-e2fd-49c5-8c89-6d94d281e3e6","Type":"ContainerStarted","Data":"a2acc945196890b495c367bc7acaf5afc5214573e95007cf842c2da728ff3979"}
Sep 08 14:56:29.113304 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:29.113228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-556b5cd57d-klghc" podStartSLOduration=33.113210855 podStartE2EDuration="33.113210855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:29.112270467 +0000 UTC m=+67.763694769" watchObservedRunningTime="2026-09-08 14:56:29.113210855 +0000 UTC m=+67.764635163"
Sep 08 14:56:29.261814 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:29.261783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:29.264757 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:29.264730    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:30.102424 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:30.102400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:30.103671 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:30.103644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-556b5cd57d-klghc"
Sep 08 14:56:32.391847 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:32.391760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:32.863537 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:32.863486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:32.965354 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:32.965320    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:33.473857 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:33.473823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:33.476332 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:33.476308    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eb0f489a-6369-42fb-8518-f1ac77cfbcb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bq2g2\" (UID: \"eb0f489a-6369-42fb-8518-f1ac77cfbcb5\") " pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:33.732818 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:33.732742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xf5qg\""
Sep 08 14:56:33.740832 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:33.740796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bq2g2"
Sep 08 14:56:35.084539 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.084482    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:35.192073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.191975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:35.273510 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.273166    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:35.382472 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.382449    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bq2g2"]
Sep 08 14:56:35.385805 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:35.385775    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0f489a_6369_42fb_8518_f1ac77cfbcb5.slice/crio-c70bf7195296bff58e5765be06e0949adfe18e784d871e039e254b27d7522de8 WatchSource:0}: Error finding container c70bf7195296bff58e5765be06e0949adfe18e784d871e039e254b27d7522de8: Status 404 returned error can't find the container with id c70bf7195296bff58e5765be06e0949adfe18e784d871e039e254b27d7522de8
Sep 08 14:56:35.680952 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.680887    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:35.781840 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.781812    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:35.924506 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.924473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:35.980124 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:35.980068    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:36.120719 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.120692    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tsw5d" event={"ID":"980b82b5-e2fd-49c5-8c89-6d94d281e3e6","Type":"ContainerStarted","Data":"07e66b07a3e5675d3713a37bf00042b9f398555e0f4fd541b1a1e7001d1b7292"}
Sep 08 14:56:36.121078 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.120811    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:56:36.122594 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.122571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kvpz" event={"ID":"6d9e2fd8-056e-49d3-a11b-70369b073da9","Type":"ContainerStarted","Data":"a9d7309c97fc77a7ec83e5cac54b26512cd6d2f1013a5f0c1d18ef27c35e2a58"}
Sep 08 14:56:36.122689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.122615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5kvpz" event={"ID":"6d9e2fd8-056e-49d3-a11b-70369b073da9","Type":"ContainerStarted","Data":"9ac7d2d21ebb93c7ae88acd0f579dd1ed1c3e17f25ae87b470ef4abb832154f5"}
Sep 08 14:56:36.124127 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.124110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq2g2" event={"ID":"eb0f489a-6369-42fb-8518-f1ac77cfbcb5","Type":"ContainerStarted","Data":"b4d430359695497fbad5598a4642c1a8eef58771027d03a2858fb8886403085a"}
Sep 08 14:56:36.124227 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.124149    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq2g2" event={"ID":"eb0f489a-6369-42fb-8518-f1ac77cfbcb5","Type":"ContainerStarted","Data":"576b2b89ddf1b8315060a24d1efa0689838f2591a7bedc2eaa935e090c10207c"}
Sep 08 14:56:36.124227 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.124169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq2g2" event={"ID":"eb0f489a-6369-42fb-8518-f1ac77cfbcb5","Type":"ContainerStarted","Data":"c70bf7195296bff58e5765be06e0949adfe18e784d871e039e254b27d7522de8"}
Sep 08 14:56:36.125898 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.125876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgn5w" event={"ID":"aff4ee34-7afb-49df-9b72-ef2e90a39866","Type":"ContainerStarted","Data":"a2a2cb08af8eb0403406ceb887a4ecd4d9f1ae4c7f46937802c48fed8df5b472"}
Sep 08 14:56:36.126002 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.125919    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zgn5w" event={"ID":"aff4ee34-7afb-49df-9b72-ef2e90a39866","Type":"ContainerStarted","Data":"55f2357a3c6a4a7ce0919c5fd8e91f10449d9794108a6a071eba60444af42e3c"}
Sep 08 14:56:36.126002 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.125947    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:36.127312 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.127288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pdmng" event={"ID":"68ff52bf-57a8-4714-a16b-b047539508d3","Type":"ContainerStarted","Data":"d4d0d2d42c5f395e384ba215930942763da567867eef2e5ddf419692a6556131"}
Sep 08 14:56:36.129280 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.129263    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt" event={"ID":"15d20d48-1773-4f82-b4cf-9463167f3e43","Type":"ContainerStarted","Data":"9d7df98df9c6a759f96c7689ffec76de94cbd9dfe86e0b34ab55fe6adc0e383d"}
Sep 08 14:56:36.129352 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.129288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt" event={"ID":"15d20d48-1773-4f82-b4cf-9463167f3e43","Type":"ContainerStarted","Data":"f85486fe6453d2ef769d484b462a08368e60bbef9bafbe53475af191070d1ca5"}
Sep 08 14:56:36.129352 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.129301    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt" event={"ID":"15d20d48-1773-4f82-b4cf-9463167f3e43","Type":"ContainerStarted","Data":"6f63dcb57fb53e1028c142a676101f85cfc2f7712839b4ec5e3b852f47fef47e"}
Sep 08 14:56:36.136406 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.136372    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tsw5d" podStartSLOduration=67.247598484 podStartE2EDuration="1m14.136361607s" totalImagesPullingTime="6.888763123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:36.135934061 +0000 UTC m=+74.787358368" watchObservedRunningTime="2026-09-08 14:56:36.136361607 +0000 UTC m=+74.787785923"
Sep 08 14:56:36.155161 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.155129    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pdmng" podStartSLOduration=66.962813288 podStartE2EDuration="1m14.155118916s" totalImagesPullingTime="7.192305628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:36.154991627 +0000 UTC m=+74.806415936" watchObservedRunningTime="2026-09-08 14:56:36.155118916 +0000 UTC m=+74.806543214"
Sep 08 14:56:36.173382 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.173342    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-78997fc645-j9vvt" podStartSLOduration=1.8335309469999999 podStartE2EDuration="9.173330329s" totalImagesPullingTime="7.339799382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:36.17284641 +0000 UTC m=+74.824270727" watchObservedRunningTime="2026-09-08 14:56:36.173330329 +0000 UTC m=+74.824754638"
Sep 08 14:56:36.188792 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.188762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5kvpz" podStartSLOduration=67.539644811 podStartE2EDuration="1m14.188752488s" totalImagesPullingTime="6.649107677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:36.188328883 +0000 UTC m=+74.839753192" watchObservedRunningTime="2026-09-08 14:56:36.188752488 +0000 UTC m=+74.840176795"
Sep 08 14:56:36.204781 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.204754    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zgn5w" podStartSLOduration=34.098596647 podStartE2EDuration="41.204744594s" totalImagesPullingTime="7.106147947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:36.204398239 +0000 UTC m=+74.855822547" watchObservedRunningTime="2026-09-08 14:56:36.204744594 +0000 UTC m=+74.856168953"
Sep 08 14:56:36.296266 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.296239    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:36.395387 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:36.395366    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:38.805486 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:38.805462    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:38.901618 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:38.901592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:39.001313 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:39.001293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:39.065358 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:39.065305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:39.141283 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:39.141259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bq2g2" event={"ID":"eb0f489a-6369-42fb-8518-f1ac77cfbcb5","Type":"ContainerStarted","Data":"d877921cef025602dbd6ff120aeeb2648c4c40c197b2952e63083f493506607a"}
Sep 08 14:56:39.159240 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:39.159195    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bq2g2" podStartSLOduration=35.226638122 podStartE2EDuration="38.159181544s" totalImagesPullingTime="2.932543422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 14:56:01 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:39.158790358 +0000 UTC m=+77.810214666" watchObservedRunningTime="2026-09-08 14:56:39.159181544 +0000 UTC m=+77.810605851"
Sep 08 14:56:41.591266 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:41.591218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:41.651334 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:41.651315    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:42.164368 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:42.164334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:42.392731 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:42.392693    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:42.443346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:42.443297    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:46.135185 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:46.135068    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zgn5w"
Sep 08 14:56:46.404171 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:46.404102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:46.503459 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:46.503427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:48.714954 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:48.714930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:48.773086 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:48.773041    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:49.202922 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:49.202900    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:49.303047 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:49.303012    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:49.650627 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:49.650601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:49.750493 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:49.750469    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:49.851980 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:49.851957    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:49.952583 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:49.952518    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:51.703670 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:51.703646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:51.762826 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:51.762804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:52.506304 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.506276    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:52.715371 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.715347    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"]
Sep 08 14:56:52.726371 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.726352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.726976 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.726958    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rgw4l"]
Sep 08 14:56:52.729085 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.729061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 08 14:56:52.729190 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.729074    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 08 14:56:52.729372 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.729352    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 14:56:52.730260 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.730231    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 14:56:52.730358 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.730272    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 14:56:52.730358 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.730320    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 08 14:56:52.730493 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.730476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qwvpz\""
Sep 08 14:56:52.734017 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.734002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"]
Sep 08 14:56:52.734116 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.734099    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.736458 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.736442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6k5q8\""
Sep 08 14:56:52.736559 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.736442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 14:56:52.736715 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.736680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 14:56:52.736715 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.736706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 14:56:52.795293 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-textfile\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795293 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795273    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-tls\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795293 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795468 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.795468 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795468 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qt6f\" (UniqueName: \"kubernetes.io/projected/9d357714-b7b0-4400-8a16-4957188c1380-kube-api-access-2qt6f\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.795468 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d357714-b7b0-4400-8a16-4957188c1380-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.795468 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9d357714-b7b0-4400-8a16-4957188c1380-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795484    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv5p6\" (UniqueName: \"kubernetes.io/projected/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-kube-api-access-wv5p6\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-wtmp\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-sys\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-metrics-client-ca\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.795689 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.795679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-root\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896097 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-wtmp\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896189 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896189 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-sys\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896189 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896203    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-metrics-client-ca\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-root\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-textfile\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896267    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-tls\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896289    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-wtmp\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-sys\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896346 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896345    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qt6f\" (UniqueName: \"kubernetes.io/projected/9d357714-b7b0-4400-8a16-4957188c1380-kube-api-access-2qt6f\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d357714-b7b0-4400-8a16-4957188c1380-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896462    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9d357714-b7b0-4400-8a16-4957188c1380-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv5p6\" (UniqueName: \"kubernetes.io/projected/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-kube-api-access-wv5p6\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896619    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-textfile\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-root\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:52.896749    2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.896785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.896960 ip-10-0-136-157 kubenswrapper[2578]: E0908 14:56:52.896809    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-tls podName:9d357714-b7b0-4400-8a16-4957188c1380 nodeName:}" failed. No retries permitted until 2026-09-08 14:56:53.396791417 +0000 UTC m=+92.048215746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-gfxpg" (UID: "9d357714-b7b0-4400-8a16-4957188c1380") : secret "kube-state-metrics-tls" not found
Sep 08 14:56:52.897489 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.897005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-metrics-client-ca\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.897608 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.897575    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9d357714-b7b0-4400-8a16-4957188c1380-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.897682 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.897612    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d357714-b7b0-4400-8a16-4957188c1380-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.897741 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.897696    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.899808 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.899778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-tls\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.899898 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.899848    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:52.900263 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.900241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.907501 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.907484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qt6f\" (UniqueName: \"kubernetes.io/projected/9d357714-b7b0-4400-8a16-4957188c1380-kube-api-access-2qt6f\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:52.907913 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:52.907897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv5p6\" (UniqueName: \"kubernetes.io/projected/6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f-kube-api-access-wv5p6\") pod \"node-exporter-rgw4l\" (UID: \"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f\") " pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:53.042836 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.042813    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgw4l"
Sep 08 14:56:53.053282 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:53.053262    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6443d6bc_b1ce_4c83_ac02_b9ec5e181c7f.slice/crio-f94cf6140f9c65e311ae9704e15f9d6f4645d589a5c42c4be40c529a134f782c WatchSource:0}: Error finding container f94cf6140f9c65e311ae9704e15f9d6f4645d589a5c42c4be40c529a134f782c: Status 404 returned error can't find the container with id f94cf6140f9c65e311ae9704e15f9d6f4645d589a5c42c4be40c529a134f782c
Sep 08 14:56:53.181835 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.181813    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgw4l" event={"ID":"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f","Type":"ContainerStarted","Data":"f94cf6140f9c65e311ae9704e15f9d6f4645d589a5c42c4be40c529a134f782c"}
Sep 08 14:56:53.207550 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.207515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:53.309097 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.309057    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:53.400380 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.400354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:53.402487 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.402469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9d357714-b7b0-4400-8a16-4957188c1380-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-gfxpg\" (UID: \"9d357714-b7b0-4400-8a16-4957188c1380\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:53.411217 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.411201    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:53.636642 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.636596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"
Sep 08 14:56:53.658392 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.658371    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:53.750140 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.750107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:53.904481 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.904416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg"]
Sep 08 14:56:53.908102 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:53.908076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d357714_b7b0_4400_8a16_4957188c1380.slice/crio-030f93f62c91c82e7e98ef81ec5fa34dc261640cac8e1803ad412379f26f1978 WatchSource:0}: Error finding container 030f93f62c91c82e7e98ef81ec5fa34dc261640cac8e1803ad412379f26f1978: Status 404 returned error can't find the container with id 030f93f62c91c82e7e98ef81ec5fa34dc261640cac8e1803ad412379f26f1978
Sep 08 14:56:53.981335 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:53.981310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:54.186125 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:54.186067    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg" event={"ID":"9d357714-b7b0-4400-8a16-4957188c1380","Type":"ContainerStarted","Data":"030f93f62c91c82e7e98ef81ec5fa34dc261640cac8e1803ad412379f26f1978"}
Sep 08 14:56:55.640738 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:55.640645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:55.698737 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:55.698673    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:56.058717 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.058694    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:56.195355 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.195326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgw4l" event={"ID":"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f","Type":"ContainerStarted","Data":"067f7a6b3166b81306adda08758e2ffcb9b35b37a814235d5d8ba2fbf3010872"}
Sep 08 14:56:56.197697 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.197664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg" event={"ID":"9d357714-b7b0-4400-8a16-4957188c1380","Type":"ContainerStarted","Data":"d28799dee6ce2025b15730840f60b00f452044be0437749af4693a012a14c216"}
Sep 08 14:56:56.197830 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.197740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg" event={"ID":"9d357714-b7b0-4400-8a16-4957188c1380","Type":"ContainerStarted","Data":"66fb2301e462259060057a50f422b79fb73eb699ea229b57d826a1e7064ce080"}
Sep 08 14:56:56.197926 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.197914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg" event={"ID":"9d357714-b7b0-4400-8a16-4957188c1380","Type":"ContainerStarted","Data":"faddf8179c535d4bcddb156c9fc1f6523937ef9f000a37d0c516893b6d669009"}
Sep 08 14:56:56.231920 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.231881    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-gfxpg" podStartSLOduration=2.8125268759999997 podStartE2EDuration="4.231866524s" totalImagesPullingTime="1.419339648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:56.231097812 +0000 UTC m=+94.882522121" watchObservedRunningTime="2026-09-08 14:56:56.231866524 +0000 UTC m=+94.883290831"
Sep 08 14:56:56.270389 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:56.270367    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:57.201956 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.201924    2578 generic.go:356] "Generic (PLEG): container finished" podID="6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f" containerID="067f7a6b3166b81306adda08758e2ffcb9b35b37a814235d5d8ba2fbf3010872" exitCode=0
Sep 08 14:56:57.202290 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.202011    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgw4l" event={"ID":"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f","Type":"ContainerDied","Data":"067f7a6b3166b81306adda08758e2ffcb9b35b37a814235d5d8ba2fbf3010872"}
Sep 08 14:56:57.485741 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.485665    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:57.579287 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.579260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:57.676947 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.676917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:57.777835 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.777819    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:57.905879 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.905857    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6947fb877c-m6t8w"]
Sep 08 14:56:57.908103 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.908089    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:57.910646 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.910597    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 08 14:56:57.910838 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.910823    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 08 14:56:57.910955 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.910900    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mb6ds\""
Sep 08 14:56:57.911189 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.911173    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 08 14:56:57.911250 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.911198    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 08 14:56:57.911250 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.911231    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 08 14:56:57.916834 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.916815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 08 14:56:57.923000 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:57.922981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6947fb877c-m6t8w"]
Sep 08 14:56:58.034166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034112    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-secret-telemeter-client\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034141    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-telemeter-client-tls\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034166 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-metrics-client-ca\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034352 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034186    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034352 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034243    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788sp\" (UniqueName: \"kubernetes.io/projected/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-kube-api-access-788sp\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034352 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-federate-client-tls\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034352 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.034471 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.034354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-serving-certs-ca-bundle\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.134923 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.134904    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-secret-telemeter-client\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135033 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.134930    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-telemeter-client-tls\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135033 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.134948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-metrics-client-ca\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135033 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.134964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135178 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788sp\" (UniqueName: \"kubernetes.io/projected/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-kube-api-access-788sp\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135178 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-federate-client-tls\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135280 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135280 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135232    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-serving-certs-ca-bundle\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135842 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135819    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135954 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-serving-certs-ca-bundle\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.135954 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.135946    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-metrics-client-ca\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.137402 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.137383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-secret-telemeter-client\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.137716 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.137695    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.137947 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.137929    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-federate-client-tls\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.137992 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.137930    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-telemeter-client-tls\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.142333 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.142314    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-788sp\" (UniqueName: \"kubernetes.io/projected/a69aa81c-fbe3-41f7-872b-c0b48851ea5e-kube-api-access-788sp\") pod \"telemeter-client-6947fb877c-m6t8w\" (UID: \"a69aa81c-fbe3-41f7-872b-c0b48851ea5e\") " pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.206931 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.206904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgw4l" event={"ID":"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f","Type":"ContainerStarted","Data":"827cdb78fd61a58fb81b63b43ee6624225bb1aba5c36a5592c691b7c0144afa3"}
Sep 08 14:56:58.207194 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.206962    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgw4l" event={"ID":"6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f","Type":"ContainerStarted","Data":"6d7acae4762617fe184833e6ee3c7f6be50856292c02125425ef75fb34abf427"}
Sep 08 14:56:58.218289 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.218266    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w"
Sep 08 14:56:58.224786 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.224750    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rgw4l" podStartSLOduration=3.159674701 podStartE2EDuration="6.224738843s" totalImagesPullingTime="2.996126379s" totalInitContainerRuntime="68.937763ms" isStatefulPod=true podCreationTimestamp="2026-09-08 14:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:56:58.224551046 +0000 UTC m=+96.875975352" watchObservedRunningTime="2026-09-08 14:56:58.224738843 +0000 UTC m=+96.876163151"
Sep 08 14:56:58.493469 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.493381    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6947fb877c-m6t8w"]
Sep 08 14:56:58.497229 ip-10-0-136-157 kubenswrapper[2578]: W0908 14:56:58.497174    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69aa81c_fbe3_41f7_872b_c0b48851ea5e.slice/crio-ac8e614260daa7f5d98c8fd2f8944496a86617de78826eee28971c97739957e4 WatchSource:0}: Error finding container ac8e614260daa7f5d98c8fd2f8944496a86617de78826eee28971c97739957e4: Status 404 returned error can't find the container with id ac8e614260daa7f5d98c8fd2f8944496a86617de78826eee28971c97739957e4
Sep 08 14:56:58.891207 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:58.891172    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:56:59.211905 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:56:59.211841    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w" event={"ID":"a69aa81c-fbe3-41f7-872b-c0b48851ea5e","Type":"ContainerStarted","Data":"ac8e614260daa7f5d98c8fd2f8944496a86617de78826eee28971c97739957e4"}
Sep 08 14:57:00.119340 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:00.119305    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:00.221318 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:00.221284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:02.223362 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:02.223293    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w" event={"ID":"a69aa81c-fbe3-41f7-872b-c0b48851ea5e","Type":"ContainerStarted","Data":"35d7c18110e5d1e4addddba9b413be00034a5799264e15ce2ae82aeb957027f7"}
Sep 08 14:57:03.228824 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:03.228744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w" event={"ID":"a69aa81c-fbe3-41f7-872b-c0b48851ea5e","Type":"ContainerStarted","Data":"5d7b5d1ab10cb6b7757a2551da45d011c25d6a0f1c34c347ca3daef2d3a79075"}
Sep 08 14:57:03.228824 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:03.228797    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w" event={"ID":"a69aa81c-fbe3-41f7-872b-c0b48851ea5e","Type":"ContainerStarted","Data":"3f6063a7e4bb828b09b681dbaeb8c0994015e4b3acf6ab6b21b18837ee4c5dee"}
Sep 08 14:57:03.250483 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:03.250438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6947fb877c-m6t8w" podStartSLOduration=1.9910857549999998 podStartE2EDuration="6.250420392s" totalImagesPullingTime="4.259334637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 14:56:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 14:57:03.249320359 +0000 UTC m=+101.900744666" watchObservedRunningTime="2026-09-08 14:57:03.250420392 +0000 UTC m=+101.901844702"
Sep 08 14:57:07.135353 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:07.135322    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tsw5d"
Sep 08 14:57:08.246824 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:08.246792    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee890970-ffe2-4317-a669-1230e0317745" containerID="17e96c270bd17639d9b86ac2aab44e8d77a0884958877000fd0b77ef6ccbcdc6" exitCode=0
Sep 08 14:57:08.247276 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:08.246873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8" event={"ID":"ee890970-ffe2-4317-a669-1230e0317745","Type":"ContainerDied","Data":"17e96c270bd17639d9b86ac2aab44e8d77a0884958877000fd0b77ef6ccbcdc6"}
Sep 08 14:57:08.247348 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:08.247282    2578 scope.go:117] "RemoveContainer" containerID="17e96c270bd17639d9b86ac2aab44e8d77a0884958877000fd0b77ef6ccbcdc6"
Sep 08 14:57:09.251891 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:09.251864    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-fnkl8" event={"ID":"ee890970-ffe2-4317-a669-1230e0317745","Type":"ContainerStarted","Data":"c155c55bfdbff9277cf3d22d7fb6152167dc58cdd07a6c5e62c0a1897db8adaf"}
Sep 08 14:57:11.965972 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:11.965933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:12.022157 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:12.022106    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:14.104988 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:14.104947    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:14.206656 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:14.206620    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:14.262214 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:14.262178    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:21.756101 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:21.756069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:23.492073 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:23.492031    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:26.632984 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:26.632955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:26.973385 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:26.973303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:28.861938 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:28.861883    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:57:30.617071 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:57:30.617046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:08.879377 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:08.879343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:08.979840 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:08.979805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:20.734994 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:20.734895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:22.615709 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:22.615674    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:22.667307 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:22.667280    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:26.974077 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:26.974038    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:27.074580 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:27.074550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:33.478905 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:33.478867    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:33.579660 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:33.579630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:58:33.679731 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:58:33.679709    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:00.708296 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:00.708254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:00.798933 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:00.798895    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:02.667378 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:02.667341    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:02.721751 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:02.721715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:37.634384 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:37.634332    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:38.458017 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:38.457982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:41.284779 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:41.284741    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:41.383884 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:41.383851    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:42.679392 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:42.679355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:42.786555 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:42.786497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:43.010752 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:43.010715    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:44.369320 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:44.369282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:44.692565 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:44.692479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:45.119857 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:45.119825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:49.424486 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:49.424451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:49.525799 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:49.525769    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:50.624519 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:50.624485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:50.725101 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:50.725069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:52.706800 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:52.706773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:55.430244 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:55.430205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:56.055451 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:56.055418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:56.150492 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:56.150464    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 14:59:59.151069 ip-10-0-136-157 kubenswrapper[2578]: I0908 14:59:59.151035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:03.167603 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:03.167568    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:03.270659 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:03.270627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:05.197323 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:05.197287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:17.438634 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.438604    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:17.536516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.536483    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"]
Sep 08 15:00:17.538624 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.538609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.544509 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.544484    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 08 15:00:17.545624 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.545597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 08 15:00:17.545702 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.545626    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 08 15:00:17.545702 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.545677    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 08 15:00:17.545801 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.545751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 08 15:00:17.545855 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.545841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4ztrb\""
Sep 08 15:00:17.552240 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.552214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"]
Sep 08 15:00:17.559077 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.559049    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:17.568126 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.568098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/472997ed-6272-40bf-bcce-6081a7240326-metrics-cert\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.568207 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.568152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/472997ed-6272-40bf-bcce-6081a7240326-cert\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.568246 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.568205    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dhv\" (UniqueName: \"kubernetes.io/projected/472997ed-6272-40bf-bcce-6081a7240326-kube-api-access-x8dhv\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.568246 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.568224    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/472997ed-6272-40bf-bcce-6081a7240326-manager-config\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.668973 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.668948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/472997ed-6272-40bf-bcce-6081a7240326-cert\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.669090 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.668981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dhv\" (UniqueName: \"kubernetes.io/projected/472997ed-6272-40bf-bcce-6081a7240326-kube-api-access-x8dhv\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.669090 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.669002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/472997ed-6272-40bf-bcce-6081a7240326-manager-config\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.669090 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.669036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/472997ed-6272-40bf-bcce-6081a7240326-metrics-cert\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.669619 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.669600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/472997ed-6272-40bf-bcce-6081a7240326-manager-config\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.671594 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.671572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/472997ed-6272-40bf-bcce-6081a7240326-cert\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.671675 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.671613    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/472997ed-6272-40bf-bcce-6081a7240326-metrics-cert\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.677906 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.677885    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dhv\" (UniqueName: \"kubernetes.io/projected/472997ed-6272-40bf-bcce-6081a7240326-kube-api-access-x8dhv\") pod \"lws-controller-manager-5c458b45b4-2h89t\" (UID: \"472997ed-6272-40bf-bcce-6081a7240326\") " pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:17.847994 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:17.847977    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:18.125773 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:18.125742    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"]
Sep 08 15:00:18.128421 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:00:18.128394    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472997ed_6272_40bf_bcce_6081a7240326.slice/crio-f987b0c5704061af1aac34cc753b694863984b1b846fb527298d8ba9b117ef0b WatchSource:0}: Error finding container f987b0c5704061af1aac34cc753b694863984b1b846fb527298d8ba9b117ef0b: Status 404 returned error can't find the container with id f987b0c5704061af1aac34cc753b694863984b1b846fb527298d8ba9b117ef0b
Sep 08 15:00:18.892148 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:18.892110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t" event={"ID":"472997ed-6272-40bf-bcce-6081a7240326","Type":"ContainerStarted","Data":"f987b0c5704061af1aac34cc753b694863984b1b846fb527298d8ba9b117ef0b"}
Sep 08 15:00:21.751016 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:21.750947    2578 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 15:00:21.903711 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:21.903565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t" event={"ID":"472997ed-6272-40bf-bcce-6081a7240326","Type":"ContainerStarted","Data":"7bf32f349b0ae01886c1da6b38daee5c931d4a619bc5c6af9ff2cf47c1b3fd9e"}
Sep 08 15:00:21.903711 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:21.903646    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:21.928005 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:21.927956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t" podStartSLOduration=1.728654219 podStartE2EDuration="4.927939726s" totalImagesPullingTime="3.199285507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:00:21.92145476 +0000 UTC m=+300.572879069" watchObservedRunningTime="2026-09-08 15:00:21.927939726 +0000 UTC m=+300.579364035"
Sep 08 15:00:22.464001 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:22.463952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:22.615626 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:22.615589    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:23.094588 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.094558    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:23.469018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.468946    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"]
Sep 08 15:00:23.471041 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.471026    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.475698 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.475677    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 15:00:23.475813 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.475710    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 15:00:23.476321 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.476307    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6bscm\""
Sep 08 15:00:23.476825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.476807    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 15:00:23.476934 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.476839    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 15:00:23.493557 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.493537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"]
Sep 08 15:00:23.510118 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.510098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aced14fe-726f-419c-91a3-eb281d7b3cb7-manifests\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.510289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.510134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aced14fe-726f-419c-91a3-eb281d7b3cb7-tmp\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.510289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.510165    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aced14fe-726f-419c-91a3-eb281d7b3cb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.510289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.510210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmt95\" (UniqueName: \"kubernetes.io/projected/aced14fe-726f-419c-91a3-eb281d7b3cb7-kube-api-access-cmt95\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.510289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.510229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aced14fe-726f-419c-91a3-eb281d7b3cb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.610836 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.610805    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aced14fe-726f-419c-91a3-eb281d7b3cb7-tmp\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.610976 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.610858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aced14fe-726f-419c-91a3-eb281d7b3cb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.610976 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.610894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmt95\" (UniqueName: \"kubernetes.io/projected/aced14fe-726f-419c-91a3-eb281d7b3cb7-kube-api-access-cmt95\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.610976 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.610918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aced14fe-726f-419c-91a3-eb281d7b3cb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.611077 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.611049    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aced14fe-726f-419c-91a3-eb281d7b3cb7-manifests\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.611393 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.611362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/aced14fe-726f-419c-91a3-eb281d7b3cb7-manifests\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.613187 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.613164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aced14fe-726f-419c-91a3-eb281d7b3cb7-tmp\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.613477 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.613461    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/aced14fe-726f-419c-91a3-eb281d7b3cb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.613555 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.613514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/aced14fe-726f-419c-91a3-eb281d7b3cb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.619193 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.619174    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmt95\" (UniqueName: \"kubernetes.io/projected/aced14fe-726f-419c-91a3-eb281d7b3cb7-kube-api-access-cmt95\") pod \"opendatahub-operator-controller-manager-6b574fc88f-6wpwh\" (UID: \"aced14fe-726f-419c-91a3-eb281d7b3cb7\") " pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:23.781823 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:23.781799    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:24.064407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:24.064322    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"]
Sep 08 15:00:24.067518 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:00:24.067478    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaced14fe_726f_419c_91a3_eb281d7b3cb7.slice/crio-ed0556a64dc9c38bd15020cfd00cc0c36ebca4c661016cb760eb261856da2d92 WatchSource:0}: Error finding container ed0556a64dc9c38bd15020cfd00cc0c36ebca4c661016cb760eb261856da2d92: Status 404 returned error can't find the container with id ed0556a64dc9c38bd15020cfd00cc0c36ebca4c661016cb760eb261856da2d92
Sep 08 15:00:24.069442 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:24.069423    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:00:24.915583 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:24.915552    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh" event={"ID":"aced14fe-726f-419c-91a3-eb281d7b3cb7","Type":"ContainerStarted","Data":"ed0556a64dc9c38bd15020cfd00cc0c36ebca4c661016cb760eb261856da2d92"}
Sep 08 15:00:25.346567 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:25.346519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:29.936147 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:29.936111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh" event={"ID":"aced14fe-726f-419c-91a3-eb281d7b3cb7","Type":"ContainerStarted","Data":"2959c23a8f67f11484736ffa93d45ff5b08dedf276b056952aead846e2b4dd5a"}
Sep 08 15:00:30.940795 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:30.940761    2578 generic.go:356] "Generic (PLEG): container finished" podID="aced14fe-726f-419c-91a3-eb281d7b3cb7" containerID="2959c23a8f67f11484736ffa93d45ff5b08dedf276b056952aead846e2b4dd5a" exitCode=0
Sep 08 15:00:30.941165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:30.940833    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh" event={"ID":"aced14fe-726f-419c-91a3-eb281d7b3cb7","Type":"ContainerDied","Data":"2959c23a8f67f11484736ffa93d45ff5b08dedf276b056952aead846e2b4dd5a"}
Sep 08 15:00:31.946194 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:31.946160    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh" event={"ID":"aced14fe-726f-419c-91a3-eb281d7b3cb7","Type":"ContainerStarted","Data":"c18f60feb046a8944d749ddcaacf4d2c6777da60b98fd7d91db01eab8c1dcd42"}
Sep 08 15:00:31.946588 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:31.946267    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:31.970289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:31.970246    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh" podStartSLOduration=2.474931176 podStartE2EDuration="8.970231901s" totalImagesPullingTime="5.484264122s" totalInitContainerRuntime="1.011036603s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:00:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:00:31.968425717 +0000 UTC m=+310.619850025" watchObservedRunningTime="2026-09-08 15:00:31.970231901 +0000 UTC m=+310.621656209"
Sep 08 15:00:32.909282 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:32.909250    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c458b45b4-2h89t"
Sep 08 15:00:34.701943 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:34.701910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:36.680881 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:36.680845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:36.783693 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:36.783659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:42.952063 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:42.952031    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b574fc88f-6wpwh"
Sep 08 15:00:45.740541 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:45.740491    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:45.846711 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:45.846681    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:46.056786 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.056760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:46.125917 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.125888    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"]
Sep 08 15:00:46.132873 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.132857    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.135740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.135719    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 15:00:46.135832 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.135821    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-grkw6\""
Sep 08 15:00:46.136560 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.136514    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 15:00:46.144280 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.144257    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"]
Sep 08 15:00:46.276759 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.276733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4d6q\" (UniqueName: \"kubernetes.io/projected/289d27f9-9d44-4881-ae48-bc4814ab9301-kube-api-access-t4d6q\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.276862 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.276768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289d27f9-9d44-4881-ae48-bc4814ab9301-tls-certs\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.276862 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.276789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/289d27f9-9d44-4881-ae48-bc4814ab9301-tmp\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.377839 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.377786    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4d6q\" (UniqueName: \"kubernetes.io/projected/289d27f9-9d44-4881-ae48-bc4814ab9301-kube-api-access-t4d6q\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.377839 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.377821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289d27f9-9d44-4881-ae48-bc4814ab9301-tls-certs\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.378003 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.377840    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/289d27f9-9d44-4881-ae48-bc4814ab9301-tmp\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.380135 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.380110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/289d27f9-9d44-4881-ae48-bc4814ab9301-tmp\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.380317 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.380301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/289d27f9-9d44-4881-ae48-bc4814ab9301-tls-certs\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.393360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.393337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4d6q\" (UniqueName: \"kubernetes.io/projected/289d27f9-9d44-4881-ae48-bc4814ab9301-kube-api-access-t4d6q\") pod \"kube-auth-proxy-5ff8c49fcd-pncgn\" (UID: \"289d27f9-9d44-4881-ae48-bc4814ab9301\") " pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.408299 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.408270    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:46.442536 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.442507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"
Sep 08 15:00:46.712903 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:46.712882    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn"]
Sep 08 15:00:46.715302 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:00:46.715276    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289d27f9_9d44_4881_ae48_bc4814ab9301.slice/crio-4ffbec50c68df260e904fba363e2950cf66993568f89a4efbad4f10940b078c1 WatchSource:0}: Error finding container 4ffbec50c68df260e904fba363e2950cf66993568f89a4efbad4f10940b078c1: Status 404 returned error can't find the container with id 4ffbec50c68df260e904fba363e2950cf66993568f89a4efbad4f10940b078c1
Sep 08 15:00:47.004785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:47.004705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn" event={"ID":"289d27f9-9d44-4881-ae48-bc4814ab9301","Type":"ContainerStarted","Data":"4ffbec50c68df260e904fba363e2950cf66993568f89a4efbad4f10940b078c1"}
Sep 08 15:00:47.617930 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:47.617890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:50.018075 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:50.018042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn" event={"ID":"289d27f9-9d44-4881-ae48-bc4814ab9301","Type":"ContainerStarted","Data":"da0cdee54e31fe0e11633c1af921c89a43eec659fd90decd0de800163ddafc03"}
Sep 08 15:00:50.034231 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:50.034186    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5ff8c49fcd-pncgn" podStartSLOduration=1.127172667 podStartE2EDuration="4.034171946s" totalImagesPullingTime="2.906999279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:00:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:00:50.033540856 +0000 UTC m=+328.684965163" watchObservedRunningTime="2026-09-08 15:00:50.034171946 +0000 UTC m=+328.685596257"
Sep 08 15:00:50.186134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:50.186102    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:50.286926 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:50.286859    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:52.459288 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:52.459254    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:52.562539 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:52.562500    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:53.236430 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:53.236398    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:53.442422 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:53.442388    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:53.539626 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:53.539594    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.104577 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.104543    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.204758 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.204730    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.305054 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.305028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.489503 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.489424    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.598662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.598631    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.708951 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.708910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:00:57.990616 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:00:57.990591    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:04.170817 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:04.170787    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:04.351283 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:04.351242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:07.533447 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:07.533411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:07.851318 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:07.851236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:08.010826 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:08.010792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:08.117045 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:08.116970    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:10.952697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:10.952662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:11.052886 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:11.052841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:40.822799 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:40.822690    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:40.923884 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:40.923841    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:41.023677 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:41.023655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:41.564819 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:41.564771    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:51.525263 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:51.525232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:51.624189 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:51.624157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:58.038684 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:58.038639    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:01:58.136712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:01:58.136669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:02:28.194735 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:02:28.194698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:09.883123 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:09.883069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:10.692479 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:10.692443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:13.571480 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:13.571438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:13.694782 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:13.694731    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:14.300309 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:14.300283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:14.697023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:14.696944    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:15.956147 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:15.956112    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:18.101255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:18.101218    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:18.230908 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:18.230871    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:20.551422 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:20.551386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:21.746504 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:21.746467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:23.767553 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:23.767499    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:23.873117 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:23.873086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:26.396964 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.396934    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"]
Sep 08 15:03:26.399744 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.399726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.402363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.402340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 15:03:26.402363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.402340    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 15:03:26.402507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.402398    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 08 15:03:26.402507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.402444    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8wkr4\""
Sep 08 15:03:26.403219 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.403200    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 08 15:03:26.403585 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.403566    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:26.408578 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.408558    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"]
Sep 08 15:03:26.482675 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.482637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/881b1609-d5f9-4823-a466-e76a62ffdc62-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.482675 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.482682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86wjp\" (UniqueName: \"kubernetes.io/projected/881b1609-d5f9-4823-a466-e76a62ffdc62-kube-api-access-86wjp\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.482810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.482793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/881b1609-d5f9-4823-a466-e76a62ffdc62-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.583765 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.583743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/881b1609-d5f9-4823-a466-e76a62ffdc62-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.583889 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.583772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86wjp\" (UniqueName: \"kubernetes.io/projected/881b1609-d5f9-4823-a466-e76a62ffdc62-kube-api-access-86wjp\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.583889 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.583835    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/881b1609-d5f9-4823-a466-e76a62ffdc62-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.584377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.584355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/881b1609-d5f9-4823-a466-e76a62ffdc62-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.586380 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.586357    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/881b1609-d5f9-4823-a466-e76a62ffdc62-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.591575 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.591555    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86wjp\" (UniqueName: \"kubernetes.io/projected/881b1609-d5f9-4823-a466-e76a62ffdc62-kube-api-access-86wjp\") pod \"kuadrant-console-plugin-598f86468b-bflpr\" (UID: \"881b1609-d5f9-4823-a466-e76a62ffdc62\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.709686 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.709626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"
Sep 08 15:03:26.979896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:26.979870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr"]
Sep 08 15:03:26.982224 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:03:26.982196    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881b1609_d5f9_4823_a466_e76a62ffdc62.slice/crio-e520265bf936c22aaa344b307039412874681c738895fcf3035d901a33b41246 WatchSource:0}: Error finding container e520265bf936c22aaa344b307039412874681c738895fcf3035d901a33b41246: Status 404 returned error can't find the container with id e520265bf936c22aaa344b307039412874681c738895fcf3035d901a33b41246
Sep 08 15:03:27.051852 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:27.051820    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:27.610884 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:27.610845    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr" event={"ID":"881b1609-d5f9-4823-a466-e76a62ffdc62","Type":"ContainerStarted","Data":"e520265bf936c22aaa344b307039412874681c738895fcf3035d901a33b41246"}
Sep 08 15:03:28.906412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:28.906382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:29.270988 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:29.270958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:29.696745 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:29.696642    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:29.797512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:29.797473    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:29.896743 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:29.896689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:29.958218 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:29.958133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:31.632207 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:31.632129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr" event={"ID":"881b1609-d5f9-4823-a466-e76a62ffdc62","Type":"ContainerStarted","Data":"e95e1e4e8de46c702dfa4f48073328ddcf9d8ab088ce60c04fb35352afb257d3"}
Sep 08 15:03:31.649314 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:31.649268    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-bflpr" podStartSLOduration=1.345663939 podStartE2EDuration="5.649252838s" totalImagesPullingTime="4.303588899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:03:31.648753218 +0000 UTC m=+490.300177527" watchObservedRunningTime="2026-09-08 15:03:31.649252838 +0000 UTC m=+490.300677146"
Sep 08 15:03:34.569649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:34.569616    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:34.670140 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:34.670114    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:36.870239 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:36.870205    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"]
Sep 08 15:03:36.873679 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:36.873660    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:36.876995 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:36.876979    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-2zmq6\""
Sep 08 15:03:36.890321 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:36.890301    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"]
Sep 08 15:03:36.969287 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:36.969263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34347ceb-52ea-4f00-bdae-0f413a0de983-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:36.969424 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:36.969374    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4cn\" (UniqueName: \"kubernetes.io/projected/34347ceb-52ea-4f00-bdae-0f413a0de983-kube-api-access-sj4cn\") pod \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.070679 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.070644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4cn\" (UniqueName: \"kubernetes.io/projected/34347ceb-52ea-4f00-bdae-0f413a0de983-kube-api-access-sj4cn\") pod \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.070785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.070700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34347ceb-52ea-4f00-bdae-0f413a0de983-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.071037 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.071022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34347ceb-52ea-4f00-bdae-0f413a0de983-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.087080 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.087055    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4cn\" (UniqueName: \"kubernetes.io/projected/34347ceb-52ea-4f00-bdae-0f413a0de983-kube-api-access-sj4cn\") pod \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.183710 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.183650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.360409 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.360372    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"]
Sep 08 15:03:37.369643 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.369611    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"]
Sep 08 15:03:37.393843 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.393819    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"]
Sep 08 15:03:37.397264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.397247    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.417214 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.417191    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"]
Sep 08 15:03:37.474031 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.473999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6595249f-7b82-4cd6-a5ad-22aa763e5a22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-6chrm\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.474163 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.474060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcx6v\" (UniqueName: \"kubernetes.io/projected/6595249f-7b82-4cd6-a5ad-22aa763e5a22-kube-api-access-mcx6v\") pod \"kuadrant-operator-controller-manager-68cf589589-6chrm\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.574927 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.574896    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcx6v\" (UniqueName: \"kubernetes.io/projected/6595249f-7b82-4cd6-a5ad-22aa763e5a22-kube-api-access-mcx6v\") pod \"kuadrant-operator-controller-manager-68cf589589-6chrm\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.575098 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.574979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6595249f-7b82-4cd6-a5ad-22aa763e5a22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-6chrm\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.575367 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.575351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6595249f-7b82-4cd6-a5ad-22aa763e5a22-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-68cf589589-6chrm\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.579088 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:03:37.579056    2578 log.go:32] "RunPodSandbox from runtime service failed" err=<
Sep 08 15:03:37.579088 ip-10-0-136-157 kubenswrapper[2578]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-xjn4v_kuadrant-system_34347ceb-52ea-4f00-bdae-0f413a0de983_0(78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-xjn4v to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0" Netns:"/var/run/netns/e0f0b387-ed60-4a50-bc41-d624361efedc" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-xjn4v;K8S_POD_INFRA_CONTAINER_ID=78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0;K8S_POD_UID=34347ceb-52ea-4f00-bdae-0f413a0de983" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v] networking: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v/34347ceb-52ea-4f00-bdae-0f413a0de983:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v 78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:13 [10.134.0.19/23]
Sep 08 15:03:37.579088 ip-10-0-136-157 kubenswrapper[2578]:         '
Sep 08 15:03:37.579088 ip-10-0-136-157 kubenswrapper[2578]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 08 15:03:37.579088 ip-10-0-136-157 kubenswrapper[2578]:  >
Sep 08 15:03:37.579260 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:03:37.579112    2578 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=<
Sep 08 15:03:37.579260 ip-10-0-136-157 kubenswrapper[2578]:         rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-68cf589589-xjn4v_kuadrant-system_34347ceb-52ea-4f00-bdae-0f413a0de983_0(78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0): error adding pod kuadrant-system_kuadrant-operator-controller-manager-68cf589589-xjn4v to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0" Netns:"/var/run/netns/e0f0b387-ed60-4a50-bc41-d624361efedc" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-68cf589589-xjn4v;K8S_POD_INFRA_CONTAINER_ID=78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0;K8S_POD_UID=34347ceb-52ea-4f00-bdae-0f413a0de983" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v] networking: [kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v/34347ceb-52ea-4f00-bdae-0f413a0de983:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v 78a74b4ed8422ca8c249511e65b4b3ec30e2a6a6f5a584111181e1fc202016b0 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:86:00:13 [10.134.0.19/23]
Sep 08 15:03:37.579260 ip-10-0-136-157 kubenswrapper[2578]:         '
Sep 08 15:03:37.579260 ip-10-0-136-157 kubenswrapper[2578]:         ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"}
Sep 08 15:03:37.579260 ip-10-0-136-157 kubenswrapper[2578]:  > pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.584535 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.584499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcx6v\" (UniqueName: \"kubernetes.io/projected/6595249f-7b82-4cd6-a5ad-22aa763e5a22-kube-api-access-mcx6v\") pod \"kuadrant-operator-controller-manager-68cf589589-6chrm\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") " pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.659602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.659577    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.661568 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:03:37.661517    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" is forbidden: User \"system:node:ip-10-0-136-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-157.ec2.internal' and this object" podUID="34347ceb-52ea-4f00-bdae-0f413a0de983" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.667422 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.667405    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.669622 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:03:37.669602    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" is forbidden: User \"system:node:ip-10-0-136-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-157.ec2.internal' and this object" podUID="34347ceb-52ea-4f00-bdae-0f413a0de983" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:37.708208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.708158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:37.776628 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.776602    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj4cn\" (UniqueName: \"kubernetes.io/projected/34347ceb-52ea-4f00-bdae-0f413a0de983-kube-api-access-sj4cn\") pod \"34347ceb-52ea-4f00-bdae-0f413a0de983\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") "
Sep 08 15:03:37.777221 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.777191    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34347ceb-52ea-4f00-bdae-0f413a0de983-extensions-socket-volume\") pod \"34347ceb-52ea-4f00-bdae-0f413a0de983\" (UID: \"34347ceb-52ea-4f00-bdae-0f413a0de983\") "
Sep 08 15:03:37.777669 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.777544    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34347ceb-52ea-4f00-bdae-0f413a0de983-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "34347ceb-52ea-4f00-bdae-0f413a0de983" (UID: "34347ceb-52ea-4f00-bdae-0f413a0de983"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:03:37.779280 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.779261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34347ceb-52ea-4f00-bdae-0f413a0de983-kube-api-access-sj4cn" (OuterVolumeSpecName: "kube-api-access-sj4cn") pod "34347ceb-52ea-4f00-bdae-0f413a0de983" (UID: "34347ceb-52ea-4f00-bdae-0f413a0de983"). InnerVolumeSpecName "kube-api-access-sj4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:03:37.830263 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.830237    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34347ceb-52ea-4f00-bdae-0f413a0de983" path="/var/lib/kubelet/pods/34347ceb-52ea-4f00-bdae-0f413a0de983/volumes"
Sep 08 15:03:37.878752 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.878725    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/34347ceb-52ea-4f00-bdae-0f413a0de983-extensions-socket-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:03:37.878752 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.878748    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj4cn\" (UniqueName: \"kubernetes.io/projected/34347ceb-52ea-4f00-bdae-0f413a0de983-kube-api-access-sj4cn\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:03:37.992501 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:37.991643    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"]
Sep 08 15:03:37.995323 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:03:37.995283    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6595249f_7b82_4cd6_a5ad_22aa763e5a22.slice/crio-3c79d12edfce8017988529fdb7b18ac9fff45dfa46b702b1ac8e62f5776587ff WatchSource:0}: Error finding container 3c79d12edfce8017988529fdb7b18ac9fff45dfa46b702b1ac8e62f5776587ff: Status 404 returned error can't find the container with id 3c79d12edfce8017988529fdb7b18ac9fff45dfa46b702b1ac8e62f5776587ff
Sep 08 15:03:38.665409 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:38.665382    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:38.665409 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:38.665384    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm" event={"ID":"6595249f-7b82-4cd6-a5ad-22aa763e5a22","Type":"ContainerStarted","Data":"3c79d12edfce8017988529fdb7b18ac9fff45dfa46b702b1ac8e62f5776587ff"}
Sep 08 15:03:38.669501 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:03:38.669470    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-68cf589589-xjn4v\" is forbidden: User \"system:node:ip-10-0-136-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-157.ec2.internal' and this object" podUID="34347ceb-52ea-4f00-bdae-0f413a0de983" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-xjn4v"
Sep 08 15:03:39.363128 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:39.363100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:39.716111 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:39.716033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:42.233383 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:42.233349    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:43.135000 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:43.134960    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:43.231829 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:43.231784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:43.932349 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:43.932306    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:44.031597 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:44.031556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:46.703231 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:46.703193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm" event={"ID":"6595249f-7b82-4cd6-a5ad-22aa763e5a22","Type":"ContainerStarted","Data":"5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898"}
Sep 08 15:03:46.703633 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:46.703329    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:46.723580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:46.723508    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm" podStartSLOduration=1.489129094 podStartE2EDuration="9.723492355s" totalImagesPullingTime="8.234363261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:03:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:03:46.722197989 +0000 UTC m=+505.373622310" watchObservedRunningTime="2026-09-08 15:03:46.723492355 +0000 UTC m=+505.374916664"
Sep 08 15:03:48.133309 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:48.133279    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:48.233374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:48.233343    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:48.590947 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:48.590910    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:48.654697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:48.654669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:48.936800 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:48.936712    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:49.001012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:49.000983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:54.402864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:54.402829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:54.461918 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:54.461886    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:57.709923 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:57.709887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:03:59.176354 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:59.176319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:03:59.326238 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:03:59.326205    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:10.105077 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.105043    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"]
Sep 08 15:04:10.105467 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.105262    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm" podUID="6595249f-7b82-4cd6-a5ad-22aa763e5a22" containerName="manager" containerID="cri-o://5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898" gracePeriod=10
Sep 08 15:04:10.356900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.356844    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:04:10.415718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.415691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcx6v\" (UniqueName: \"kubernetes.io/projected/6595249f-7b82-4cd6-a5ad-22aa763e5a22-kube-api-access-mcx6v\") pod \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") "
Sep 08 15:04:10.415815 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.415725    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6595249f-7b82-4cd6-a5ad-22aa763e5a22-extensions-socket-volume\") pod \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\" (UID: \"6595249f-7b82-4cd6-a5ad-22aa763e5a22\") "
Sep 08 15:04:10.416197 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.416176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6595249f-7b82-4cd6-a5ad-22aa763e5a22-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6595249f-7b82-4cd6-a5ad-22aa763e5a22" (UID: "6595249f-7b82-4cd6-a5ad-22aa763e5a22"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:04:10.418011 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.417993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6595249f-7b82-4cd6-a5ad-22aa763e5a22-kube-api-access-mcx6v" (OuterVolumeSpecName: "kube-api-access-mcx6v") pod "6595249f-7b82-4cd6-a5ad-22aa763e5a22" (UID: "6595249f-7b82-4cd6-a5ad-22aa763e5a22"). InnerVolumeSpecName "kube-api-access-mcx6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:04:10.516869 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.516832    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcx6v\" (UniqueName: \"kubernetes.io/projected/6595249f-7b82-4cd6-a5ad-22aa763e5a22-kube-api-access-mcx6v\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:10.516972 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.516874    2578 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6595249f-7b82-4cd6-a5ad-22aa763e5a22-extensions-socket-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:10.805740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.805708    2578 generic.go:356] "Generic (PLEG): container finished" podID="6595249f-7b82-4cd6-a5ad-22aa763e5a22" containerID="5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898" exitCode=0
Sep 08 15:04:10.805864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.805773    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"
Sep 08 15:04:10.805864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.805795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm" event={"ID":"6595249f-7b82-4cd6-a5ad-22aa763e5a22","Type":"ContainerDied","Data":"5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898"}
Sep 08 15:04:10.805961 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.805884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm" event={"ID":"6595249f-7b82-4cd6-a5ad-22aa763e5a22","Type":"ContainerDied","Data":"3c79d12edfce8017988529fdb7b18ac9fff45dfa46b702b1ac8e62f5776587ff"}
Sep 08 15:04:10.805961 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.805909    2578 scope.go:117] "RemoveContainer" containerID="5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898"
Sep 08 15:04:10.815605 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.815584    2578 scope.go:117] "RemoveContainer" containerID="5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898"
Sep 08 15:04:10.815956 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:04:10.815935    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898\": container with ID starting with 5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898 not found: ID does not exist" containerID="5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898"
Sep 08 15:04:10.816004 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.815965    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898"} err="failed to get container status \"5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898\": rpc error: code = NotFound desc = could not find container \"5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898\": container with ID starting with 5212109037ee1ee25a3fc754968f0be9f19617cca2e6cfab578d33d0fe110898 not found: ID does not exist"
Sep 08 15:04:10.834636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.834612    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"]
Sep 08 15:04:10.840018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:10.839994    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-68cf589589-6chrm"]
Sep 08 15:04:11.830474 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:11.830446    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6595249f-7b82-4cd6-a5ad-22aa763e5a22" path="/var/lib/kubelet/pods/6595249f-7b82-4cd6-a5ad-22aa763e5a22/volumes"
Sep 08 15:04:14.229699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:14.229621    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:14.341486 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:14.341459    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:15.258469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:15.258434    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:17.996383 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:17.996347    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:17.996736 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:17.996660    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6595249f-7b82-4cd6-a5ad-22aa763e5a22" containerName="manager"
Sep 08 15:04:17.996736 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:17.996675    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6595249f-7b82-4cd6-a5ad-22aa763e5a22" containerName="manager"
Sep 08 15:04:17.996736 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:17.996729    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6595249f-7b82-4cd6-a5ad-22aa763e5a22" containerName="manager"
Sep 08 15:04:17.998913 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:17.998896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:17.999879 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:17.999858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:18.002497 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.002474    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 15:04:18.009050 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.009027    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:18.069798 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.069773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hgn\" (UniqueName: \"kubernetes.io/projected/d009cac7-8178-4813-bbb2-a8ed897e5066-kube-api-access-h4hgn\") pod \"limitador-limitador-859c4d96d6-zpfh4\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.069929 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.069903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d009cac7-8178-4813-bbb2-a8ed897e5066-config-file\") pod \"limitador-limitador-859c4d96d6-zpfh4\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.099199 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.099176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:18.104668 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.104643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:18.170751 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.170732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d009cac7-8178-4813-bbb2-a8ed897e5066-config-file\") pod \"limitador-limitador-859c4d96d6-zpfh4\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.170854 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.170771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hgn\" (UniqueName: \"kubernetes.io/projected/d009cac7-8178-4813-bbb2-a8ed897e5066-kube-api-access-h4hgn\") pod \"limitador-limitador-859c4d96d6-zpfh4\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.171318 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.171303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d009cac7-8178-4813-bbb2-a8ed897e5066-config-file\") pod \"limitador-limitador-859c4d96d6-zpfh4\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.178411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.178388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hgn\" (UniqueName: \"kubernetes.io/projected/d009cac7-8178-4813-bbb2-a8ed897e5066-kube-api-access-h4hgn\") pod \"limitador-limitador-859c4d96d6-zpfh4\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.191094 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.191074    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:18.308922 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.308886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:18.580281 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.580220    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:18.583364 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:18.583338    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd009cac7_8178_4813_bbb2_a8ed897e5066.slice/crio-c8faefdf4e981d228bb47fc3af0459ea58f4e4dc52d52b7abbb483c2aea6b524 WatchSource:0}: Error finding container c8faefdf4e981d228bb47fc3af0459ea58f4e4dc52d52b7abbb483c2aea6b524: Status 404 returned error can't find the container with id c8faefdf4e981d228bb47fc3af0459ea58f4e4dc52d52b7abbb483c2aea6b524
Sep 08 15:04:18.788274 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.788249    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-mmg4w"]
Sep 08 15:04:18.791125 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.791105    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:18.793429 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.793412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-vfmph\""
Sep 08 15:04:18.798310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.798277    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mmg4w"]
Sep 08 15:04:18.836015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.835961    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:18.844271 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.844251    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4" event={"ID":"d009cac7-8178-4813-bbb2-a8ed897e5066","Type":"ContainerStarted","Data":"c8faefdf4e981d228bb47fc3af0459ea58f4e4dc52d52b7abbb483c2aea6b524"}
Sep 08 15:04:18.877724 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.877702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfhzk\" (UniqueName: \"kubernetes.io/projected/79a6a47c-6e1d-4e7e-b868-856dc2f6d667-kube-api-access-gfhzk\") pod \"authorino-b54df94b4-mmg4w\" (UID: \"79a6a47c-6e1d-4e7e-b868-856dc2f6d667\") " pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:18.917484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.917458    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:18.920957 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.920936    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-pjvsn"]
Sep 08 15:04:18.923132 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.923115    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:18.929298 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.929278    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-pjvsn"]
Sep 08 15:04:18.978066 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.978041    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/81fd2bd0-c8fe-4a15-95d4-868be358f3f3-kube-api-access-2vwzw\") pod \"authorino-c548b5d7d-pjvsn\" (UID: \"81fd2bd0-c8fe-4a15-95d4-868be358f3f3\") " pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:18.978184 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.978167    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfhzk\" (UniqueName: \"kubernetes.io/projected/79a6a47c-6e1d-4e7e-b868-856dc2f6d667-kube-api-access-gfhzk\") pod \"authorino-b54df94b4-mmg4w\" (UID: \"79a6a47c-6e1d-4e7e-b868-856dc2f6d667\") " pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:18.985933 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:18.985914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfhzk\" (UniqueName: \"kubernetes.io/projected/79a6a47c-6e1d-4e7e-b868-856dc2f6d667-kube-api-access-gfhzk\") pod \"authorino-b54df94b4-mmg4w\" (UID: \"79a6a47c-6e1d-4e7e-b868-856dc2f6d667\") " pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:19.078694 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.078671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/81fd2bd0-c8fe-4a15-95d4-868be358f3f3-kube-api-access-2vwzw\") pod \"authorino-c548b5d7d-pjvsn\" (UID: \"81fd2bd0-c8fe-4a15-95d4-868be358f3f3\") " pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:19.088644 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.088598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/81fd2bd0-c8fe-4a15-95d4-868be358f3f3-kube-api-access-2vwzw\") pod \"authorino-c548b5d7d-pjvsn\" (UID: \"81fd2bd0-c8fe-4a15-95d4-868be358f3f3\") " pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:19.100948 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.100927    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:19.231377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.231347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:19.368887 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.368820    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mmg4w"]
Sep 08 15:04:19.372144 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:19.372118    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79a6a47c_6e1d_4e7e_b868_856dc2f6d667.slice/crio-cd8ee999a70c3e6399db049f9d18f1b766ec20877b3afc9223a6021a07f5f3da WatchSource:0}: Error finding container cd8ee999a70c3e6399db049f9d18f1b766ec20877b3afc9223a6021a07f5f3da: Status 404 returned error can't find the container with id cd8ee999a70c3e6399db049f9d18f1b766ec20877b3afc9223a6021a07f5f3da
Sep 08 15:04:19.496344 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.496323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-pjvsn"]
Sep 08 15:04:19.498719 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:19.498690    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81fd2bd0_c8fe_4a15_95d4_868be358f3f3.slice/crio-e999696f926456bd47ae17594393c6c6ee1907a2954c248428938d4920529a21 WatchSource:0}: Error finding container e999696f926456bd47ae17594393c6c6ee1907a2954c248428938d4920529a21: Status 404 returned error can't find the container with id e999696f926456bd47ae17594393c6c6ee1907a2954c248428938d4920529a21
Sep 08 15:04:19.849391 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.849366    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-pjvsn" event={"ID":"81fd2bd0-c8fe-4a15-95d4-868be358f3f3","Type":"ContainerStarted","Data":"e999696f926456bd47ae17594393c6c6ee1907a2954c248428938d4920529a21"}
Sep 08 15:04:19.850579 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:19.850558    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mmg4w" event={"ID":"79a6a47c-6e1d-4e7e-b868-856dc2f6d667","Type":"ContainerStarted","Data":"cd8ee999a70c3e6399db049f9d18f1b766ec20877b3afc9223a6021a07f5f3da"}
Sep 08 15:04:22.299651 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:22.299619    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:22.870511 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:22.870478    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mmg4w" event={"ID":"79a6a47c-6e1d-4e7e-b868-856dc2f6d667","Type":"ContainerStarted","Data":"3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2"}
Sep 08 15:04:22.872016 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:22.871993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-pjvsn" event={"ID":"81fd2bd0-c8fe-4a15-95d4-868be358f3f3","Type":"ContainerStarted","Data":"e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a"}
Sep 08 15:04:22.885888 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:22.885847    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-mmg4w" podStartSLOduration=2.414864943 podStartE2EDuration="4.885835713s" totalImagesPullingTime="2.47097077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:04:22.885402905 +0000 UTC m=+541.536827213" watchObservedRunningTime="2026-09-08 15:04:22.885835713 +0000 UTC m=+541.537260022"
Sep 08 15:04:22.903213 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:22.903172    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-pjvsn" podStartSLOduration=2.567829893 podStartE2EDuration="4.903159333s" totalImagesPullingTime="2.33532944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:04:22.902088522 +0000 UTC m=+541.553512843" watchObservedRunningTime="2026-09-08 15:04:22.903159333 +0000 UTC m=+541.554583642"
Sep 08 15:04:22.927358 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:22.927336    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mmg4w"]
Sep 08 15:04:24.883473 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:24.883437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4" event={"ID":"d009cac7-8178-4813-bbb2-a8ed897e5066","Type":"ContainerStarted","Data":"389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e"}
Sep 08 15:04:24.883934 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:24.883477    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-mmg4w" podUID="79a6a47c-6e1d-4e7e-b868-856dc2f6d667" containerName="authorino" containerID="cri-o://3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2" gracePeriod=30
Sep 08 15:04:24.913445 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:24.913395    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4" podStartSLOduration=2.253796498 podStartE2EDuration="7.913379583s" totalImagesPullingTime="5.659583085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:04:24.912574895 +0000 UTC m=+543.563999207" watchObservedRunningTime="2026-09-08 15:04:24.913379583 +0000 UTC m=+543.564803892"
Sep 08 15:04:25.112802 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.112774    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:25.227488 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.227419    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfhzk\" (UniqueName: \"kubernetes.io/projected/79a6a47c-6e1d-4e7e-b868-856dc2f6d667-kube-api-access-gfhzk\") pod \"79a6a47c-6e1d-4e7e-b868-856dc2f6d667\" (UID: \"79a6a47c-6e1d-4e7e-b868-856dc2f6d667\") "
Sep 08 15:04:25.229541 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.229491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a6a47c-6e1d-4e7e-b868-856dc2f6d667-kube-api-access-gfhzk" (OuterVolumeSpecName: "kube-api-access-gfhzk") pod "79a6a47c-6e1d-4e7e-b868-856dc2f6d667" (UID: "79a6a47c-6e1d-4e7e-b868-856dc2f6d667"). InnerVolumeSpecName "kube-api-access-gfhzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:04:25.328172 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.328141    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfhzk\" (UniqueName: \"kubernetes.io/projected/79a6a47c-6e1d-4e7e-b868-856dc2f6d667-kube-api-access-gfhzk\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:25.888579 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.888544    2578 generic.go:356] "Generic (PLEG): container finished" podID="79a6a47c-6e1d-4e7e-b868-856dc2f6d667" containerID="3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2" exitCode=0
Sep 08 15:04:25.888953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.888587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-mmg4w"
Sep 08 15:04:25.888953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.888625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mmg4w" event={"ID":"79a6a47c-6e1d-4e7e-b868-856dc2f6d667","Type":"ContainerDied","Data":"3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2"}
Sep 08 15:04:25.888953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.888668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-mmg4w" event={"ID":"79a6a47c-6e1d-4e7e-b868-856dc2f6d667","Type":"ContainerDied","Data":"cd8ee999a70c3e6399db049f9d18f1b766ec20877b3afc9223a6021a07f5f3da"}
Sep 08 15:04:25.888953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.888686    2578 scope.go:117] "RemoveContainer" containerID="3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2"
Sep 08 15:04:25.888953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.888935    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:25.897707 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.897690    2578 scope.go:117] "RemoveContainer" containerID="3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2"
Sep 08 15:04:25.898025 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:04:25.897996    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2\": container with ID starting with 3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2 not found: ID does not exist" containerID="3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2"
Sep 08 15:04:25.898136 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.898033    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2"} err="failed to get container status \"3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2\": rpc error: code = NotFound desc = could not find container \"3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2\": container with ID starting with 3e5dc89b8b55f3997cb3a6d563c748080e6481a4becd37d937c58529eeaab5d2 not found: ID does not exist"
Sep 08 15:04:25.903423 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.903390    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mmg4w"]
Sep 08 15:04:25.905110 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:25.905085    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-mmg4w"]
Sep 08 15:04:27.831283 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:27.831243    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79a6a47c-6e1d-4e7e-b868-856dc2f6d667" path="/var/lib/kubelet/pods/79a6a47c-6e1d-4e7e-b868-856dc2f6d667/volumes"
Sep 08 15:04:32.650767 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:32.650729    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:32.651224 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:32.650952    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4" podUID="d009cac7-8178-4813-bbb2-a8ed897e5066" containerName="limitador" containerID="cri-o://389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e" gracePeriod=30
Sep 08 15:04:32.652957 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:32.652934    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:33.585367 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.585347    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:33.684044 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.684022    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4hgn\" (UniqueName: \"kubernetes.io/projected/d009cac7-8178-4813-bbb2-a8ed897e5066-kube-api-access-h4hgn\") pod \"d009cac7-8178-4813-bbb2-a8ed897e5066\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") "
Sep 08 15:04:33.684341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.684097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d009cac7-8178-4813-bbb2-a8ed897e5066-config-file\") pod \"d009cac7-8178-4813-bbb2-a8ed897e5066\" (UID: \"d009cac7-8178-4813-bbb2-a8ed897e5066\") "
Sep 08 15:04:33.684401 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.684381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d009cac7-8178-4813-bbb2-a8ed897e5066-config-file" (OuterVolumeSpecName: "config-file") pod "d009cac7-8178-4813-bbb2-a8ed897e5066" (UID: "d009cac7-8178-4813-bbb2-a8ed897e5066"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:04:33.686202 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.686185    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d009cac7-8178-4813-bbb2-a8ed897e5066-kube-api-access-h4hgn" (OuterVolumeSpecName: "kube-api-access-h4hgn") pod "d009cac7-8178-4813-bbb2-a8ed897e5066" (UID: "d009cac7-8178-4813-bbb2-a8ed897e5066"). InnerVolumeSpecName "kube-api-access-h4hgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:04:33.785469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.785449    2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d009cac7-8178-4813-bbb2-a8ed897e5066-config-file\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:33.785469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.785470    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hgn\" (UniqueName: \"kubernetes.io/projected/d009cac7-8178-4813-bbb2-a8ed897e5066-kube-api-access-h4hgn\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:33.923979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.923949    2578 generic.go:356] "Generic (PLEG): container finished" podID="d009cac7-8178-4813-bbb2-a8ed897e5066" containerID="389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e" exitCode=0
Sep 08 15:04:33.924085 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.923986    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"
Sep 08 15:04:33.924085 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.923999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4" event={"ID":"d009cac7-8178-4813-bbb2-a8ed897e5066","Type":"ContainerDied","Data":"389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e"}
Sep 08 15:04:33.924085 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.924029    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-zpfh4" event={"ID":"d009cac7-8178-4813-bbb2-a8ed897e5066","Type":"ContainerDied","Data":"c8faefdf4e981d228bb47fc3af0459ea58f4e4dc52d52b7abbb483c2aea6b524"}
Sep 08 15:04:33.924085 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.924050    2578 scope.go:117] "RemoveContainer" containerID="389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e"
Sep 08 15:04:33.931913 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.931897    2578 scope.go:117] "RemoveContainer" containerID="389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e"
Sep 08 15:04:33.932190 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:04:33.932170    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e\": container with ID starting with 389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e not found: ID does not exist" containerID="389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e"
Sep 08 15:04:33.932253 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.932198    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e"} err="failed to get container status \"389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e\": rpc error: code = NotFound desc = could not find container \"389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e\": container with ID starting with 389a2b87a38b6e4020e296a79ee5c21e3edae086bf033c673953a9adee0f0e8e not found: ID does not exist"
Sep 08 15:04:33.939266 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.939247    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:33.943547 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:33.943510    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-zpfh4"]
Sep 08 15:04:35.830900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:35.830864    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d009cac7-8178-4813-bbb2-a8ed897e5066" path="/var/lib/kubelet/pods/d009cac7-8178-4813-bbb2-a8ed897e5066/volumes"
Sep 08 15:04:37.904509 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:37.904477    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:40.554691 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:40.554659    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:40.614456 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:40.614433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:50.672764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.672725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:50.913104 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913071    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xwsjs"]
Sep 08 15:04:50.913370 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913353    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79a6a47c-6e1d-4e7e-b868-856dc2f6d667" containerName="authorino"
Sep 08 15:04:50.913417 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913375    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="79a6a47c-6e1d-4e7e-b868-856dc2f6d667" containerName="authorino"
Sep 08 15:04:50.913417 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913398    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d009cac7-8178-4813-bbb2-a8ed897e5066" containerName="limitador"
Sep 08 15:04:50.913417 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913403    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d009cac7-8178-4813-bbb2-a8ed897e5066" containerName="limitador"
Sep 08 15:04:50.913504 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913453    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="79a6a47c-6e1d-4e7e-b868-856dc2f6d667" containerName="authorino"
Sep 08 15:04:50.913504 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.913460    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d009cac7-8178-4813-bbb2-a8ed897e5066" containerName="limitador"
Sep 08 15:04:50.915516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.915502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:50.924514 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:50.924460    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xwsjs"]
Sep 08 15:04:51.004030 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.004003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vzw\" (UniqueName: \"kubernetes.io/projected/99930208-599e-4047-a304-188743cc5a65-kube-api-access-57vzw\") pod \"authorino-5dc7ccb5f-xwsjs\" (UID: \"99930208-599e-4047-a304-188743cc5a65\") " pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:51.104760 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.104736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vzw\" (UniqueName: \"kubernetes.io/projected/99930208-599e-4047-a304-188743cc5a65-kube-api-access-57vzw\") pod \"authorino-5dc7ccb5f-xwsjs\" (UID: \"99930208-599e-4047-a304-188743cc5a65\") " pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:51.114250 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.114227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vzw\" (UniqueName: \"kubernetes.io/projected/99930208-599e-4047-a304-188743cc5a65-kube-api-access-57vzw\") pod \"authorino-5dc7ccb5f-xwsjs\" (UID: \"99930208-599e-4047-a304-188743cc5a65\") " pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:51.144714 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.144689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:51.177646 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.177596    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xwsjs"]
Sep 08 15:04:51.177788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.177778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:51.198354 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.198302    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74dc57c99f-lnhds"]
Sep 08 15:04:51.200659 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.200642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:51.210658 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.210639    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74dc57c99f-lnhds"]
Sep 08 15:04:51.237619 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.237591    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74dc57c99f-lnhds"]
Sep 08 15:04:51.237838 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:04:51.237819    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6q6xg], unattached volumes=[], failed to process volumes=[kube-api-access-6q6xg]: context canceled" pod="kuadrant-system/authorino-74dc57c99f-lnhds" podUID="c0c1cc1d-0a40-474d-aeb9-a79e89810383"
Sep 08 15:04:51.265663 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.265638    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59df484ff7-lplr6"]
Sep 08 15:04:51.267699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.267684    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.270102 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.270082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 08 15:04:51.275713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.275694    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59df484ff7-lplr6"]
Sep 08 15:04:51.307234 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.307214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c0c1cc1d-0a40-474d-aeb9-a79e89810383-kube-api-access-6q6xg\") pod \"authorino-74dc57c99f-lnhds\" (UID: \"c0c1cc1d-0a40-474d-aeb9-a79e89810383\") " pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:51.408088 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.408060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/017466ce-2a12-483b-926c-ac6d0c4413ba-tls-cert\") pod \"authorino-59df484ff7-lplr6\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") " pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.408192 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.408100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c0c1cc1d-0a40-474d-aeb9-a79e89810383-kube-api-access-6q6xg\") pod \"authorino-74dc57c99f-lnhds\" (UID: \"c0c1cc1d-0a40-474d-aeb9-a79e89810383\") " pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:51.408192 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.408124    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhrk\" (UniqueName: \"kubernetes.io/projected/017466ce-2a12-483b-926c-ac6d0c4413ba-kube-api-access-pnhrk\") pod \"authorino-59df484ff7-lplr6\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") " pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.416689 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.416667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c0c1cc1d-0a40-474d-aeb9-a79e89810383-kube-api-access-6q6xg\") pod \"authorino-74dc57c99f-lnhds\" (UID: \"c0c1cc1d-0a40-474d-aeb9-a79e89810383\") " pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:51.449948 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.449896    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xwsjs"]
Sep 08 15:04:51.452885 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:51.452865    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99930208_599e_4047_a304_188743cc5a65.slice/crio-f4a39329e38ffc2118bbba8c12a04e77dcffed507cdc6f57cda99396ddf9a327 WatchSource:0}: Error finding container f4a39329e38ffc2118bbba8c12a04e77dcffed507cdc6f57cda99396ddf9a327: Status 404 returned error can't find the container with id f4a39329e38ffc2118bbba8c12a04e77dcffed507cdc6f57cda99396ddf9a327
Sep 08 15:04:51.508954 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.508934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/017466ce-2a12-483b-926c-ac6d0c4413ba-tls-cert\") pod \"authorino-59df484ff7-lplr6\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") " pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.509047 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.508965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhrk\" (UniqueName: \"kubernetes.io/projected/017466ce-2a12-483b-926c-ac6d0c4413ba-kube-api-access-pnhrk\") pod \"authorino-59df484ff7-lplr6\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") " pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.511265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.511243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/017466ce-2a12-483b-926c-ac6d0c4413ba-tls-cert\") pod \"authorino-59df484ff7-lplr6\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") " pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.525275 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.525253    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhrk\" (UniqueName: \"kubernetes.io/projected/017466ce-2a12-483b-926c-ac6d0c4413ba-kube-api-access-pnhrk\") pod \"authorino-59df484ff7-lplr6\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") " pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.576783 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.576763    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:04:51.845851 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.845822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59df484ff7-lplr6"]
Sep 08 15:04:51.849592 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:51.849569    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod017466ce_2a12_483b_926c_ac6d0c4413ba.slice/crio-7372dec78d4110c5cbc5fd96834fdb12d4c4f22e6ae3bf62078dd3d315914f46 WatchSource:0}: Error finding container 7372dec78d4110c5cbc5fd96834fdb12d4c4f22e6ae3bf62078dd3d315914f46: Status 404 returned error can't find the container with id 7372dec78d4110c5cbc5fd96834fdb12d4c4f22e6ae3bf62078dd3d315914f46
Sep 08 15:04:51.996573 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.996540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59df484ff7-lplr6" event={"ID":"017466ce-2a12-483b-926c-ac6d0c4413ba","Type":"ContainerStarted","Data":"7372dec78d4110c5cbc5fd96834fdb12d4c4f22e6ae3bf62078dd3d315914f46"}
Sep 08 15:04:51.997699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.997675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs" event={"ID":"99930208-599e-4047-a304-188743cc5a65","Type":"ContainerStarted","Data":"f4a39329e38ffc2118bbba8c12a04e77dcffed507cdc6f57cda99396ddf9a327"}
Sep 08 15:04:51.997699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:51.997693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:52.003310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:52.003292    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:52.112883 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:52.112817    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c0c1cc1d-0a40-474d-aeb9-a79e89810383-kube-api-access-6q6xg\") pod \"c0c1cc1d-0a40-474d-aeb9-a79e89810383\" (UID: \"c0c1cc1d-0a40-474d-aeb9-a79e89810383\") "
Sep 08 15:04:52.115161 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:52.115128    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c1cc1d-0a40-474d-aeb9-a79e89810383-kube-api-access-6q6xg" (OuterVolumeSpecName: "kube-api-access-6q6xg") pod "c0c1cc1d-0a40-474d-aeb9-a79e89810383" (UID: "c0c1cc1d-0a40-474d-aeb9-a79e89810383"). InnerVolumeSpecName "kube-api-access-6q6xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:04:52.214032 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:52.214006    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q6xg\" (UniqueName: \"kubernetes.io/projected/c0c1cc1d-0a40-474d-aeb9-a79e89810383-kube-api-access-6q6xg\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:53.003474 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.003437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59df484ff7-lplr6" event={"ID":"017466ce-2a12-483b-926c-ac6d0c4413ba","Type":"ContainerStarted","Data":"6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf"}
Sep 08 15:04:53.005011 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.004991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74dc57c99f-lnhds"
Sep 08 15:04:53.005146 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.005011    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs" podUID="99930208-599e-4047-a304-188743cc5a65" containerName="authorino" containerID="cri-o://c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e" gracePeriod=30
Sep 08 15:04:53.005146 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.004987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs" event={"ID":"99930208-599e-4047-a304-188743cc5a65","Type":"ContainerStarted","Data":"c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e"}
Sep 08 15:04:53.023025 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.022985    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59df484ff7-lplr6" podStartSLOduration=1.657118987 podStartE2EDuration="2.022972817s" totalImagesPullingTime="365.85383ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:04:53.022284603 +0000 UTC m=+571.673708913" watchObservedRunningTime="2026-09-08 15:04:53.022972817 +0000 UTC m=+571.674397126"
Sep 08 15:04:53.051344 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.051316    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-74dc57c99f-lnhds"]
Sep 08 15:04:53.054308 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.054284    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-pjvsn"]
Sep 08 15:04:53.054485 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.054461    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-pjvsn" podUID="81fd2bd0-c8fe-4a15-95d4-868be358f3f3" containerName="authorino" containerID="cri-o://e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a" gracePeriod=30
Sep 08 15:04:53.059010 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.058975    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-74dc57c99f-lnhds"]
Sep 08 15:04:53.077012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.076637    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs" podStartSLOduration=2.5263386580000002 podStartE2EDuration="3.076619724s" totalImagesPullingTime="550.281066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:04:53.075183178 +0000 UTC m=+571.726607488" watchObservedRunningTime="2026-09-08 15:04:53.076619724 +0000 UTC m=+571.728044034"
Sep 08 15:04:53.308632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.308611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:53.311121 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.311103    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:53.424462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.424433    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-57vzw\" (UniqueName: \"kubernetes.io/projected/99930208-599e-4047-a304-188743cc5a65-kube-api-access-57vzw\") pod \"99930208-599e-4047-a304-188743cc5a65\" (UID: \"99930208-599e-4047-a304-188743cc5a65\") "
Sep 08 15:04:53.424603 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.424483    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/81fd2bd0-c8fe-4a15-95d4-868be358f3f3-kube-api-access-2vwzw\") pod \"81fd2bd0-c8fe-4a15-95d4-868be358f3f3\" (UID: \"81fd2bd0-c8fe-4a15-95d4-868be358f3f3\") "
Sep 08 15:04:53.426603 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.426581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fd2bd0-c8fe-4a15-95d4-868be358f3f3-kube-api-access-2vwzw" (OuterVolumeSpecName: "kube-api-access-2vwzw") pod "81fd2bd0-c8fe-4a15-95d4-868be358f3f3" (UID: "81fd2bd0-c8fe-4a15-95d4-868be358f3f3"). InnerVolumeSpecName "kube-api-access-2vwzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:04:53.426695 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.426609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99930208-599e-4047-a304-188743cc5a65-kube-api-access-57vzw" (OuterVolumeSpecName: "kube-api-access-57vzw") pod "99930208-599e-4047-a304-188743cc5a65" (UID: "99930208-599e-4047-a304-188743cc5a65"). InnerVolumeSpecName "kube-api-access-57vzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:04:53.525502 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.525478    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vzw\" (UniqueName: \"kubernetes.io/projected/99930208-599e-4047-a304-188743cc5a65-kube-api-access-57vzw\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:53.525502 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.525500    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vwzw\" (UniqueName: \"kubernetes.io/projected/81fd2bd0-c8fe-4a15-95d4-868be358f3f3-kube-api-access-2vwzw\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:04:53.830987 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:53.830960    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0c1cc1d-0a40-474d-aeb9-a79e89810383" path="/var/lib/kubelet/pods/c0c1cc1d-0a40-474d-aeb9-a79e89810383/volumes"
Sep 08 15:04:54.010666 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.010633    2578 generic.go:356] "Generic (PLEG): container finished" podID="99930208-599e-4047-a304-188743cc5a65" containerID="c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e" exitCode=0
Sep 08 15:04:54.011063 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.010679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs"
Sep 08 15:04:54.011063 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.010719    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs" event={"ID":"99930208-599e-4047-a304-188743cc5a65","Type":"ContainerDied","Data":"c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e"}
Sep 08 15:04:54.011063 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.010765    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-xwsjs" event={"ID":"99930208-599e-4047-a304-188743cc5a65","Type":"ContainerDied","Data":"f4a39329e38ffc2118bbba8c12a04e77dcffed507cdc6f57cda99396ddf9a327"}
Sep 08 15:04:54.011063 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.010790    2578 scope.go:117] "RemoveContainer" containerID="c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e"
Sep 08 15:04:54.012156 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.012126    2578 generic.go:356] "Generic (PLEG): container finished" podID="81fd2bd0-c8fe-4a15-95d4-868be358f3f3" containerID="e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a" exitCode=0
Sep 08 15:04:54.012254 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.012160    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-pjvsn"
Sep 08 15:04:54.012254 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.012224    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-pjvsn" event={"ID":"81fd2bd0-c8fe-4a15-95d4-868be358f3f3","Type":"ContainerDied","Data":"e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a"}
Sep 08 15:04:54.012360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.012260    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-pjvsn" event={"ID":"81fd2bd0-c8fe-4a15-95d4-868be358f3f3","Type":"ContainerDied","Data":"e999696f926456bd47ae17594393c6c6ee1907a2954c248428938d4920529a21"}
Sep 08 15:04:54.018666 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.018651    2578 scope.go:117] "RemoveContainer" containerID="c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e"
Sep 08 15:04:54.018883 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:04:54.018866    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e\": container with ID starting with c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e not found: ID does not exist" containerID="c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e"
Sep 08 15:04:54.018922 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.018892    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e"} err="failed to get container status \"c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e\": rpc error: code = NotFound desc = could not find container \"c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e\": container with ID starting with c2e3f208c72eabe30b7fde59aa362183a358030fe477a484c22e5ea27afd588e not found: ID does not exist"
Sep 08 15:04:54.018922 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.018911    2578 scope.go:117] "RemoveContainer" containerID="e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a"
Sep 08 15:04:54.026265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.026226    2578 scope.go:117] "RemoveContainer" containerID="e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a"
Sep 08 15:04:54.026562 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:04:54.026543    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a\": container with ID starting with e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a not found: ID does not exist" containerID="e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a"
Sep 08 15:04:54.026622 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.026568    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a"} err="failed to get container status \"e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a\": rpc error: code = NotFound desc = could not find container \"e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a\": container with ID starting with e3414d2bd312c13a8c04973814b51f9c2e58f42e933f75075555352c75f4386a not found: ID does not exist"
Sep 08 15:04:54.027839 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.027821    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-pjvsn"]
Sep 08 15:04:54.032381 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.032353    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-pjvsn"]
Sep 08 15:04:54.043621 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.043604    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xwsjs"]
Sep 08 15:04:54.048095 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.048076    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-xwsjs"]
Sep 08 15:04:54.387393 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:54.387355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:55.830578 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:55.830516    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fd2bd0-c8fe-4a15-95d4-868be358f3f3" path="/var/lib/kubelet/pods/81fd2bd0-c8fe-4a15-95d4-868be358f3f3/volumes"
Sep 08 15:04:55.830947 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:55.830847    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99930208-599e-4047-a304-188743cc5a65" path="/var/lib/kubelet/pods/99930208-599e-4047-a304-188743cc5a65/volumes"
Sep 08 15:04:58.218044 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.218016    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:58.264469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264444    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-854fb857c7-b588h"]
Sep 08 15:04:58.264741 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264728    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99930208-599e-4047-a304-188743cc5a65" containerName="authorino"
Sep 08 15:04:58.264783 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264744    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99930208-599e-4047-a304-188743cc5a65" containerName="authorino"
Sep 08 15:04:58.264783 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264758    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fd2bd0-c8fe-4a15-95d4-868be358f3f3" containerName="authorino"
Sep 08 15:04:58.264783 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264763    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fd2bd0-c8fe-4a15-95d4-868be358f3f3" containerName="authorino"
Sep 08 15:04:58.264880 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264817    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fd2bd0-c8fe-4a15-95d4-868be358f3f3" containerName="authorino"
Sep 08 15:04:58.264880 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.264827    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99930208-599e-4047-a304-188743cc5a65" containerName="authorino"
Sep 08 15:04:58.269801 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.269783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.272412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.272384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dv4gq\""
Sep 08 15:04:58.272584 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.272428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 15:04:58.272742 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.272729    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 15:04:58.273040 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.273027    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 15:04:58.285471 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.285451    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-854fb857c7-b588h"]
Sep 08 15:04:58.318226 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.318204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:58.365916 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.365889    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-webhook-cert\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.366032 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.365931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8flcb\" (UniqueName: \"kubernetes.io/projected/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-kube-api-access-8flcb\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.366032 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.365960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-metrics-certs\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.418597 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.418567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:58.451599 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.451572    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-6lcrg"]
Sep 08 15:04:58.455337 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.455319    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.463878 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.463861    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-6lcrg"]
Sep 08 15:04:58.467083 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.467066    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-metrics-certs\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.467183 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.467107    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-webhook-cert\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.467183 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.467138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8flcb\" (UniqueName: \"kubernetes.io/projected/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-kube-api-access-8flcb\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.469756 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.469711    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-webhook-cert\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.469832 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.469779    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-metrics-certs\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.474837 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.474812    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8flcb\" (UniqueName: \"kubernetes.io/projected/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-kube-api-access-8flcb\") pod \"maas-controller-854fb857c7-b588h\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") " pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.499298 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.499278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:04:58.568329 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.568307    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wws7x\" (UniqueName: \"kubernetes.io/projected/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-kube-api-access-wws7x\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.568410 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.568372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-metrics-certs\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.568410 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.568389    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-webhook-cert\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.580295 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.580277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:04:58.669256 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.669224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-metrics-certs\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.669256 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.669257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-webhook-cert\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.669400 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.669288    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wws7x\" (UniqueName: \"kubernetes.io/projected/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-kube-api-access-wws7x\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.671646 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.671625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-metrics-certs\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.671742 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.671659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-webhook-cert\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.677324 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.677302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wws7x\" (UniqueName: \"kubernetes.io/projected/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-kube-api-access-wws7x\") pod \"maas-controller-5fb695bbcb-6lcrg\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") " pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.765900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.765874    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:04:58.849608 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:58.849587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-854fb857c7-b588h"]
Sep 08 15:04:58.851447 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:58.851409    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86bad80_c7f8_422d_bec9_d2a1a11b3fb7.slice/crio-08b9282802099bad6057f8341975d0384f0db1a4089b639ec2be3ca22f47727f WatchSource:0}: Error finding container 08b9282802099bad6057f8341975d0384f0db1a4089b639ec2be3ca22f47727f: Status 404 returned error can't find the container with id 08b9282802099bad6057f8341975d0384f0db1a4089b639ec2be3ca22f47727f
Sep 08 15:04:59.036311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:59.036246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-854fb857c7-b588h" event={"ID":"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7","Type":"ContainerStarted","Data":"08b9282802099bad6057f8341975d0384f0db1a4089b639ec2be3ca22f47727f"}
Sep 08 15:04:59.036846 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:04:59.036817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-6lcrg"]
Sep 08 15:04:59.038736 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:04:59.038712    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd71a30_d5bb_4c2a_bb52_a54161d6494f.slice/crio-833b85ee87384edebec4936879eac3c24347a5e411777763268c43ee25d07f6b WatchSource:0}: Error finding container 833b85ee87384edebec4936879eac3c24347a5e411777763268c43ee25d07f6b: Status 404 returned error can't find the container with id 833b85ee87384edebec4936879eac3c24347a5e411777763268c43ee25d07f6b
Sep 08 15:05:00.046035 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:00.045955    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg" event={"ID":"1bd71a30-d5bb-4c2a-bb52-a54161d6494f","Type":"ContainerStarted","Data":"833b85ee87384edebec4936879eac3c24347a5e411777763268c43ee25d07f6b"}
Sep 08 15:05:01.771912 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:01.771877    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:01.880737 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:01.880666    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:02.055686 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:02.055646    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-854fb857c7-b588h" event={"ID":"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7","Type":"ContainerStarted","Data":"6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a"}
Sep 08 15:05:02.055863 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:02.055747    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:05:02.057152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:02.057122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg" event={"ID":"1bd71a30-d5bb-4c2a-bb52-a54161d6494f","Type":"ContainerStarted","Data":"c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926"}
Sep 08 15:05:02.057274 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:02.057265    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:05:02.075771 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:02.075728    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-854fb857c7-b588h" podStartSLOduration=1.353880325 podStartE2EDuration="4.075715428s" totalImagesPullingTime="2.721835103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:05:02.073962997 +0000 UTC m=+580.725387310" watchObservedRunningTime="2026-09-08 15:05:02.075715428 +0000 UTC m=+580.727139737"
Sep 08 15:05:02.093015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:02.092965    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg" podStartSLOduration=1.545753041 podStartE2EDuration="4.092948215s" totalImagesPullingTime="2.547195174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:04:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:05:02.091897263 +0000 UTC m=+580.743321572" watchObservedRunningTime="2026-09-08 15:05:02.092948215 +0000 UTC m=+580.744372525"
Sep 08 15:05:13.063221 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.063188    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:05:13.063690 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.063263    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:05:13.125937 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.125903    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-854fb857c7-b588h"]
Sep 08 15:05:13.126201 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.126170    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-854fb857c7-b588h" podUID="b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" containerName="manager" containerID="cri-o://6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a" gracePeriod=30
Sep 08 15:05:13.370140 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.370118    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:05:13.385399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.385371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-webhook-cert\") pod \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") "
Sep 08 15:05:13.385559 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.385420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8flcb\" (UniqueName: \"kubernetes.io/projected/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-kube-api-access-8flcb\") pod \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") "
Sep 08 15:05:13.387918 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.387880    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-kube-api-access-8flcb" (OuterVolumeSpecName: "kube-api-access-8flcb") pod "b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" (UID: "b86bad80-c7f8-422d-bec9-d2a1a11b3fb7"). InnerVolumeSpecName "kube-api-access-8flcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:05:13.388127 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.388102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" (UID: "b86bad80-c7f8-422d-bec9-d2a1a11b3fb7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:05:13.486353 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.486323    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-metrics-certs\") pod \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\" (UID: \"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7\") "
Sep 08 15:05:13.486503 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.486486    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-webhook-cert\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:05:13.486503 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.486498    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8flcb\" (UniqueName: \"kubernetes.io/projected/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-kube-api-access-8flcb\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:05:13.488399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.488376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" (UID: "b86bad80-c7f8-422d-bec9-d2a1a11b3fb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:05:13.587440 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:13.587407    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:05:14.109426 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.109331    2578 generic.go:356] "Generic (PLEG): container finished" podID="b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" containerID="6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a" exitCode=0
Sep 08 15:05:14.109426 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.109375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-854fb857c7-b588h"
Sep 08 15:05:14.109876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.109418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-854fb857c7-b588h" event={"ID":"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7","Type":"ContainerDied","Data":"6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a"}
Sep 08 15:05:14.109876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.109471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-854fb857c7-b588h" event={"ID":"b86bad80-c7f8-422d-bec9-d2a1a11b3fb7","Type":"ContainerDied","Data":"08b9282802099bad6057f8341975d0384f0db1a4089b639ec2be3ca22f47727f"}
Sep 08 15:05:14.109876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.109495    2578 scope.go:117] "RemoveContainer" containerID="6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a"
Sep 08 15:05:14.117921 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.117903    2578 scope.go:117] "RemoveContainer" containerID="6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a"
Sep 08 15:05:14.118178 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:05:14.118160    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a\": container with ID starting with 6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a not found: ID does not exist" containerID="6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a"
Sep 08 15:05:14.118227 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.118186    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a"} err="failed to get container status \"6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a\": rpc error: code = NotFound desc = could not find container \"6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a\": container with ID starting with 6b9135d1ba7f54405023d7cf0fe2d6f4a436b7a15e09db188565538044d6691a not found: ID does not exist"
Sep 08 15:05:14.126197 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.126171    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-854fb857c7-b588h"]
Sep 08 15:05:14.130088 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:14.130066    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-854fb857c7-b588h"]
Sep 08 15:05:15.831398 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:15.831362    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" path="/var/lib/kubelet/pods/b86bad80-c7f8-422d-bec9-d2a1a11b3fb7/volumes"
Sep 08 15:05:33.466993 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:33.466958    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:33.568971 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:33.568936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:34.864090 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:34.864051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:35.160818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.160736    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"]
Sep 08 15:05:35.161041 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.161028    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" containerName="manager"
Sep 08 15:05:35.161109 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.161042    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" containerName="manager"
Sep 08 15:05:35.161109 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.161100    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86bad80-c7f8-422d-bec9-d2a1a11b3fb7" containerName="manager"
Sep 08 15:05:35.167180 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.167158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.169724 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.169706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 15:05:35.170743 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.170718    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-857r2\""
Sep 08 15:05:35.170743 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.170727    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 15:05:35.170929 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.170800    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 15:05:35.170929 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.170869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 15:05:35.173633 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.173610    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"]
Sep 08 15:05:35.249019 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.248996    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjzzz\" (UniqueName: \"kubernetes.io/projected/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-kube-api-access-wjzzz\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.249134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.249039    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-maas-api-tls\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.249134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.249062    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-openshift-service-ca\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.249220 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.249127    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-metrics-certs\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.350303 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.350279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-maas-api-tls\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.350403 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.350311    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-openshift-service-ca\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.350403 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.350338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-metrics-certs\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.350403 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.350373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjzzz\" (UniqueName: \"kubernetes.io/projected/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-kube-api-access-wjzzz\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.350965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.350945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-openshift-service-ca\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.352910 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.352884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-maas-api-tls\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.352994 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.352943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-metrics-certs\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.358894 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.358873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjzzz\" (UniqueName: \"kubernetes.io/projected/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-kube-api-access-wjzzz\") pod \"maas-api-54bb5c4896-xpxqr\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") " pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.479416 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.479364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:35.770336 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.770305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"]
Sep 08 15:05:35.775117 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:05:35.775091    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5a9c75_92d5_4d5f_89b1_eb563eec232d.slice/crio-dab1f1327c277e7fab86300ddd0e528dfeeb84e8770dd6fe91d21de3d8b26084 WatchSource:0}: Error finding container dab1f1327c277e7fab86300ddd0e528dfeeb84e8770dd6fe91d21de3d8b26084: Status 404 returned error can't find the container with id dab1f1327c277e7fab86300ddd0e528dfeeb84e8770dd6fe91d21de3d8b26084
Sep 08 15:05:35.777101 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:35.777078    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:05:36.200126 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:36.200044    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr" event={"ID":"bb5a9c75-92d5-4d5f-89b1-eb563eec232d","Type":"ContainerStarted","Data":"dab1f1327c277e7fab86300ddd0e528dfeeb84e8770dd6fe91d21de3d8b26084"}
Sep 08 15:05:38.211967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:38.211929    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr" event={"ID":"bb5a9c75-92d5-4d5f-89b1-eb563eec232d","Type":"ContainerStarted","Data":"b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851"}
Sep 08 15:05:38.212395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:38.212086    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:38.230084 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:38.230033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr" podStartSLOduration=1.621855431 podStartE2EDuration="3.230020435s" totalImagesPullingTime="1.608165004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:05:38.228781465 +0000 UTC m=+616.880205776" watchObservedRunningTime="2026-09-08 15:05:38.230020435 +0000 UTC m=+616.881444744"
Sep 08 15:05:39.490496 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:39.490467    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:44.222570 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:44.222470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:05:45.305642 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:45.305607    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:45.410240 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:45.410210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:05:51.432307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:05:51.432277    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:06.272714 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.272684    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"]
Sep 08 15:06:06.276197 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.276175    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.297365 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.297347    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-mw2sm\""
Sep 08 15:06:06.298262 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.298244    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 15:06:06.299386 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.299366    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 15:06:06.301184 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.301165    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"]
Sep 08 15:06:06.336664 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.336646    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 08 15:06:06.367844 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.367822    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.367944 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.367851    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.367944 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.367881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.367944 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.367899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.368064 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.367962    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.368064 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.368001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zxkb\" (UniqueName: \"kubernetes.io/projected/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-kube-api-access-7zxkb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.421949 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.421922    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:06.468402 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zxkb\" (UniqueName: \"kubernetes.io/projected/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-kube-api-access-7zxkb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468499 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468499 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468499 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468455    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468499 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468477    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468742 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468514    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.468983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.468907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.470822 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.470804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.471149 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.471132    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.494468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.494444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zxkb\" (UniqueName: \"kubernetes.io/projected/77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20-kube-api-access-7zxkb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-zz9j6\" (UID: \"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.586654 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.586598    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:06.860637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:06.860550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"]
Sep 08 15:06:06.863239 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:06:06.863215    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e96791_e9d1_4fa6_b8e9_0d82c8a2ab20.slice/crio-525256a16bfe6ba5b9cd01824196db068a1bc6bae3a733ded9703eb9bc38152f WatchSource:0}: Error finding container 525256a16bfe6ba5b9cd01824196db068a1bc6bae3a733ded9703eb9bc38152f: Status 404 returned error can't find the container with id 525256a16bfe6ba5b9cd01824196db068a1bc6bae3a733ded9703eb9bc38152f
Sep 08 15:06:07.332209 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:07.332159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6" event={"ID":"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20","Type":"ContainerStarted","Data":"525256a16bfe6ba5b9cd01824196db068a1bc6bae3a733ded9703eb9bc38152f"}
Sep 08 15:06:07.469821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:07.469786    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:08.337971 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:08.337939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6" event={"ID":"77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20","Type":"ContainerStarted","Data":"e5cb005ff2aa6ddc37956c0750d24a805ac8575fcaf217aec9fa9da8bb13e5b8"}
Sep 08 15:06:08.369453 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:08.369405    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6" podStartSLOduration=1.079265589 podStartE2EDuration="2.369389832s" totalImagesPullingTime="1.290124243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:06:08.367250003 +0000 UTC m=+647.018674311" watchObservedRunningTime="2026-09-08 15:06:08.369389832 +0000 UTC m=+647.020814141"
Sep 08 15:06:16.587070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:16.587034    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:16.587556 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:16.587102    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:16.599280 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:16.599256    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:17.031705 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:17.031672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:17.386400 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:17.386336    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-zz9j6"
Sep 08 15:06:23.688468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.688430    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"]
Sep 08 15:06:23.691787 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.691766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.694704 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.694677    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 08 15:06:23.701761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.701735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"]
Sep 08 15:06:23.797772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.797742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.797901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.797789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9052627c-ec6d-4133-add5-c4d3bf33f324-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.797901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.797820    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.797901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.797852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgqzv\" (UniqueName: \"kubernetes.io/projected/9052627c-ec6d-4133-add5-c4d3bf33f324-kube-api-access-wgqzv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.798027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.797918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.798027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.797961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898603    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9052627c-ec6d-4133-add5-c4d3bf33f324-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898620    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898819 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgqzv\" (UniqueName: \"kubernetes.io/projected/9052627c-ec6d-4133-add5-c4d3bf33f324-kube-api-access-wgqzv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898910 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898890    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.898990 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.899042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.898995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.900913 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.900892    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9052627c-ec6d-4133-add5-c4d3bf33f324-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.901070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.901051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9052627c-ec6d-4133-add5-c4d3bf33f324-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:23.906826 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:23.906810    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgqzv\" (UniqueName: \"kubernetes.io/projected/9052627c-ec6d-4133-add5-c4d3bf33f324-kube-api-access-wgqzv\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qcdpt\" (UID: \"9052627c-ec6d-4133-add5-c4d3bf33f324\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:24.004997 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:24.004928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:24.275402 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:24.275344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"]
Sep 08 15:06:24.277575 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:06:24.277550    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9052627c_ec6d_4133_add5_c4d3bf33f324.slice/crio-0ddd2d1df63e3c24dda02d14e6144dcfa197b5c1229eda8d6e7e4bf92f027fd5 WatchSource:0}: Error finding container 0ddd2d1df63e3c24dda02d14e6144dcfa197b5c1229eda8d6e7e4bf92f027fd5: Status 404 returned error can't find the container with id 0ddd2d1df63e3c24dda02d14e6144dcfa197b5c1229eda8d6e7e4bf92f027fd5
Sep 08 15:06:24.408066 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:24.408040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt" event={"ID":"9052627c-ec6d-4133-add5-c4d3bf33f324","Type":"ContainerStarted","Data":"0ddd2d1df63e3c24dda02d14e6144dcfa197b5c1229eda8d6e7e4bf92f027fd5"}
Sep 08 15:06:25.414756 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:25.414718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt" event={"ID":"9052627c-ec6d-4133-add5-c4d3bf33f324","Type":"ContainerStarted","Data":"9572ca5010dae856a3b8aacf964d5d2e8b52bb54ed5e1acb479d215b45a58f40"}
Sep 08 15:06:25.435964 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:25.435914    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt" podStartSLOduration=1.846467948 podStartE2EDuration="2.435897749s" totalImagesPullingTime="589.429801ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:06:25.433959998 +0000 UTC m=+664.085384307" watchObservedRunningTime="2026-09-08 15:06:25.435897749 +0000 UTC m=+664.087322058"
Sep 08 15:06:26.868236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:26.868202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:34.005642 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:34.005597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:34.006039 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:34.005663    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:34.018068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:34.018041    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:34.464437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:34.464404    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qcdpt"
Sep 08 15:06:44.126013 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:44.125975    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:58.519722 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.519685    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"]
Sep 08 15:06:58.522338 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.522321    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.525455 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.525432    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 15:06:58.540772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.540752    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"]
Sep 08 15:06:58.549534 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.549495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rldr2\" (UniqueName: \"kubernetes.io/projected/6f633d2f-f332-4f43-90bf-a51a9ddca42c-kube-api-access-rldr2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.549635 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.549569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f633d2f-f332-4f43-90bf-a51a9ddca42c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.549635 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.549620    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.549744 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.549679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.549744 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.549701    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.549744 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.549719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.553815 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.553796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:06:58.650848 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.650823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rldr2\" (UniqueName: \"kubernetes.io/projected/6f633d2f-f332-4f43-90bf-a51a9ddca42c-kube-api-access-rldr2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.650955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.650863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f633d2f-f332-4f43-90bf-a51a9ddca42c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.650955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.650882    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.650955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.650913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.650955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.650938    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.650955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.650955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.651370 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.651340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.651370 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.651366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.651506 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.651392    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.653219 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.653199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f633d2f-f332-4f43-90bf-a51a9ddca42c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.653299 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.653287    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f633d2f-f332-4f43-90bf-a51a9ddca42c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.662451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.662434    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rldr2\" (UniqueName: \"kubernetes.io/projected/6f633d2f-f332-4f43-90bf-a51a9ddca42c-kube-api-access-rldr2\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-k4cl2\" (UID: \"6f633d2f-f332-4f43-90bf-a51a9ddca42c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:58.832896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:58.832841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:06:59.105762 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:59.105735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"]
Sep 08 15:06:59.108489 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:06:59.108458    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f633d2f_f332_4f43_90bf_a51a9ddca42c.slice/crio-6a3585bce7b6ab02946bb645ef7534728cc5c43e4c72cef626486b4f8b81a62b WatchSource:0}: Error finding container 6a3585bce7b6ab02946bb645ef7534728cc5c43e4c72cef626486b4f8b81a62b: Status 404 returned error can't find the container with id 6a3585bce7b6ab02946bb645ef7534728cc5c43e4c72cef626486b4f8b81a62b
Sep 08 15:06:59.559096 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:59.559062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2" event={"ID":"6f633d2f-f332-4f43-90bf-a51a9ddca42c","Type":"ContainerStarted","Data":"fb6edc3368c35f1182e9cccf69624b6d0842f9d459515297aec70804afa09c73"}
Sep 08 15:06:59.559433 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:59.559122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2" event={"ID":"6f633d2f-f332-4f43-90bf-a51a9ddca42c","Type":"ContainerStarted","Data":"6a3585bce7b6ab02946bb645ef7534728cc5c43e4c72cef626486b4f8b81a62b"}
Sep 08 15:06:59.577844 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:06:59.577801    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2" podStartSLOduration=1.403963333 podStartE2EDuration="1.577785788s" totalImagesPullingTime="173.822455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:06:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:06:59.576170282 +0000 UTC m=+698.227594594" watchObservedRunningTime="2026-09-08 15:06:59.577785788 +0000 UTC m=+698.229210097"
Sep 08 15:07:08.833799 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:08.833764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:07:08.834257 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:08.833825    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:07:08.846264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:08.846243    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:07:09.613959 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:09.613932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-k4cl2"
Sep 08 15:07:13.486788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.486712    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"]
Sep 08 15:07:13.490178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.490147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.492974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.492954    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 08 15:07:13.501462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.501444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"]
Sep 08 15:07:13.527254 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.527232    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:07:13.560771 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.560732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.560967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.560794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.560967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.560847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.560967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.560869    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d5e28c-117a-4aeb-a036-c552b5886f92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.560967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.560892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.561155 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.561022    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s86rt\" (UniqueName: \"kubernetes.io/projected/70d5e28c-117a-4aeb-a036-c552b5886f92-kube-api-access-s86rt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.661904 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.661868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.661904 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.661903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d5e28c-117a-4aeb-a036-c552b5886f92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662162 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.661921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662162 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.661970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s86rt\" (UniqueName: \"kubernetes.io/projected/70d5e28c-117a-4aeb-a036-c552b5886f92-kube-api-access-s86rt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662162 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.661998    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662162 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.662022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662392 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.662369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662442 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.662406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.662442 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.662422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.664511 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.664488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70d5e28c-117a-4aeb-a036-c552b5886f92-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.664846 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.664811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70d5e28c-117a-4aeb-a036-c552b5886f92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.681508 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.681485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s86rt\" (UniqueName: \"kubernetes.io/projected/70d5e28c-117a-4aeb-a036-c552b5886f92-kube-api-access-s86rt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-k2clf\" (UID: \"70d5e28c-117a-4aeb-a036-c552b5886f92\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:13.801389 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:13.801360    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:14.078498 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:14.078427    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"]
Sep 08 15:07:14.081614 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:07:14.081588    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70d5e28c_117a_4aeb_a036_c552b5886f92.slice/crio-20259f177edf85f8cb8b76029bf1197335acd099eb1c01a91ffe634183b095c5 WatchSource:0}: Error finding container 20259f177edf85f8cb8b76029bf1197335acd099eb1c01a91ffe634183b095c5: Status 404 returned error can't find the container with id 20259f177edf85f8cb8b76029bf1197335acd099eb1c01a91ffe634183b095c5
Sep 08 15:07:14.626316 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:14.626228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf" event={"ID":"70d5e28c-117a-4aeb-a036-c552b5886f92","Type":"ContainerStarted","Data":"4fd600516cef79c0ac14d4c92d2dd4149816209af7808c15bfb315aa01036357"}
Sep 08 15:07:14.626316 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:14.626266    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf" event={"ID":"70d5e28c-117a-4aeb-a036-c552b5886f92","Type":"ContainerStarted","Data":"20259f177edf85f8cb8b76029bf1197335acd099eb1c01a91ffe634183b095c5"}
Sep 08 15:07:14.647491 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:14.647445    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf" podStartSLOduration=1.42132783 podStartE2EDuration="1.647430083s" totalImagesPullingTime="226.102253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:07:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:07:14.646149304 +0000 UTC m=+713.297573614" watchObservedRunningTime="2026-09-08 15:07:14.647430083 +0000 UTC m=+713.298854391"
Sep 08 15:07:23.801777 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:23.801745    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:23.802149 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:23.801793    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:23.813712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:23.813686    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:24.680872 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:24.680843    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-k2clf"
Sep 08 15:07:54.039081 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:07:54.039051    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:09.135150 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.135107    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-645df54cd8-jx685"]
Sep 08 15:08:09.138327 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.138308    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.149927 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.149904    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-645df54cd8-jx685"]
Sep 08 15:08:09.287451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.287428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82v5x\" (UniqueName: \"kubernetes.io/projected/c4e14ee6-fe20-4687-9525-e9d7e411b745-kube-api-access-82v5x\") pod \"authorino-645df54cd8-jx685\" (UID: \"c4e14ee6-fe20-4687-9525-e9d7e411b745\") " pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.287585 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.287463    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4e14ee6-fe20-4687-9525-e9d7e411b745-tls-cert\") pod \"authorino-645df54cd8-jx685\" (UID: \"c4e14ee6-fe20-4687-9525-e9d7e411b745\") " pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.388883 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.388813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82v5x\" (UniqueName: \"kubernetes.io/projected/c4e14ee6-fe20-4687-9525-e9d7e411b745-kube-api-access-82v5x\") pod \"authorino-645df54cd8-jx685\" (UID: \"c4e14ee6-fe20-4687-9525-e9d7e411b745\") " pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.388883 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.388869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4e14ee6-fe20-4687-9525-e9d7e411b745-tls-cert\") pod \"authorino-645df54cd8-jx685\" (UID: \"c4e14ee6-fe20-4687-9525-e9d7e411b745\") " pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.391512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.391488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4e14ee6-fe20-4687-9525-e9d7e411b745-tls-cert\") pod \"authorino-645df54cd8-jx685\" (UID: \"c4e14ee6-fe20-4687-9525-e9d7e411b745\") " pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.398774 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.398752    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82v5x\" (UniqueName: \"kubernetes.io/projected/c4e14ee6-fe20-4687-9525-e9d7e411b745-kube-api-access-82v5x\") pod \"authorino-645df54cd8-jx685\" (UID: \"c4e14ee6-fe20-4687-9525-e9d7e411b745\") " pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.447451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.447432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-645df54cd8-jx685"
Sep 08 15:08:09.718023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.718002    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-645df54cd8-jx685"]
Sep 08 15:08:09.720461 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:08:09.720433    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4e14ee6_fe20_4687_9525_e9d7e411b745.slice/crio-4b2cd4db2af6203bb40484b3a287b5f73dac6f1c05b72f534192c3224421c93c WatchSource:0}: Error finding container 4b2cd4db2af6203bb40484b3a287b5f73dac6f1c05b72f534192c3224421c93c: Status 404 returned error can't find the container with id 4b2cd4db2af6203bb40484b3a287b5f73dac6f1c05b72f534192c3224421c93c
Sep 08 15:08:09.864770 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:09.864744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-645df54cd8-jx685" event={"ID":"c4e14ee6-fe20-4687-9525-e9d7e411b745","Type":"ContainerStarted","Data":"4b2cd4db2af6203bb40484b3a287b5f73dac6f1c05b72f534192c3224421c93c"}
Sep 08 15:08:10.871625 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:10.871579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-645df54cd8-jx685" event={"ID":"c4e14ee6-fe20-4687-9525-e9d7e411b745","Type":"ContainerStarted","Data":"dadc1114e393e1be9d80a1c70fe2385e89f432def80ba0bf90c470639bc543d8"}
Sep 08 15:08:10.888979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:10.888906    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-645df54cd8-jx685" podStartSLOduration=1.274116171 podStartE2EDuration="1.888886198s" totalImagesPullingTime="614.770027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:08:10.886439389 +0000 UTC m=+769.537863700" watchObservedRunningTime="2026-09-08 15:08:10.888886198 +0000 UTC m=+769.540310508"
Sep 08 15:08:10.917437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:10.917403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59df484ff7-lplr6"]
Sep 08 15:08:10.917663 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:10.917637    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59df484ff7-lplr6" podUID="017466ce-2a12-483b-926c-ac6d0c4413ba" containerName="authorino" containerID="cri-o://6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf" gracePeriod=30
Sep 08 15:08:11.183720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.183698    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:08:11.204092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.204066    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnhrk\" (UniqueName: \"kubernetes.io/projected/017466ce-2a12-483b-926c-ac6d0c4413ba-kube-api-access-pnhrk\") pod \"017466ce-2a12-483b-926c-ac6d0c4413ba\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") "
Sep 08 15:08:11.204230 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.204149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/017466ce-2a12-483b-926c-ac6d0c4413ba-tls-cert\") pod \"017466ce-2a12-483b-926c-ac6d0c4413ba\" (UID: \"017466ce-2a12-483b-926c-ac6d0c4413ba\") "
Sep 08 15:08:11.206789 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.206730    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017466ce-2a12-483b-926c-ac6d0c4413ba-kube-api-access-pnhrk" (OuterVolumeSpecName: "kube-api-access-pnhrk") pod "017466ce-2a12-483b-926c-ac6d0c4413ba" (UID: "017466ce-2a12-483b-926c-ac6d0c4413ba"). InnerVolumeSpecName "kube-api-access-pnhrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:08:11.215210 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.215187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017466ce-2a12-483b-926c-ac6d0c4413ba-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "017466ce-2a12-483b-926c-ac6d0c4413ba" (UID: "017466ce-2a12-483b-926c-ac6d0c4413ba"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:08:11.304793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.304755    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/017466ce-2a12-483b-926c-ac6d0c4413ba-tls-cert\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:08:11.304793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.304787    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhrk\" (UniqueName: \"kubernetes.io/projected/017466ce-2a12-483b-926c-ac6d0c4413ba-kube-api-access-pnhrk\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:08:11.497312 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.497207    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:11.607354 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.607321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:11.878227 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.878193    2578 generic.go:356] "Generic (PLEG): container finished" podID="017466ce-2a12-483b-926c-ac6d0c4413ba" containerID="6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf" exitCode=0
Sep 08 15:08:11.878623 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.878242    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59df484ff7-lplr6"
Sep 08 15:08:11.878623 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.878290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59df484ff7-lplr6" event={"ID":"017466ce-2a12-483b-926c-ac6d0c4413ba","Type":"ContainerDied","Data":"6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf"}
Sep 08 15:08:11.878623 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.878333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59df484ff7-lplr6" event={"ID":"017466ce-2a12-483b-926c-ac6d0c4413ba","Type":"ContainerDied","Data":"7372dec78d4110c5cbc5fd96834fdb12d4c4f22e6ae3bf62078dd3d315914f46"}
Sep 08 15:08:11.878623 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.878357    2578 scope.go:117] "RemoveContainer" containerID="6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf"
Sep 08 15:08:11.888456 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.888436    2578 scope.go:117] "RemoveContainer" containerID="6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf"
Sep 08 15:08:11.888779 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:08:11.888759    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf\": container with ID starting with 6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf not found: ID does not exist" containerID="6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf"
Sep 08 15:08:11.888880 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.888794    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf"} err="failed to get container status \"6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf\": rpc error: code = NotFound desc = could not find container \"6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf\": container with ID starting with 6f1a4b12b7d1db0e7ed937925c9309c5c59d0fa25d4eadda4cc1200a9d79cbdf not found: ID does not exist"
Sep 08 15:08:11.893701 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.893678    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59df484ff7-lplr6"]
Sep 08 15:08:11.897459 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:11.897441    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59df484ff7-lplr6"]
Sep 08 15:08:12.616310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:12.616268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:12.707233 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:12.707199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:13.831865 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:13.831826    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017466ce-2a12-483b-926c-ac6d0c4413ba" path="/var/lib/kubelet/pods/017466ce-2a12-483b-926c-ac6d0c4413ba/volumes"
Sep 08 15:08:28.242261 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:28.242231    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:28.431629 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:28.431601    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:28.695036 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:28.695006    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:29.066305 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:29.066275    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:29.759173 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:29.759136    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:29.855650 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:29.855618    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:30.264626 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.264592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:30.286756 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.286731    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-140253605945136"]
Sep 08 15:08:30.287045 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.287034    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017466ce-2a12-483b-926c-ac6d0c4413ba" containerName="authorino"
Sep 08 15:08:30.287094 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.287047    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="017466ce-2a12-483b-926c-ac6d0c4413ba" containerName="authorino"
Sep 08 15:08:30.287136 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.287104    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="017466ce-2a12-483b-926c-ac6d0c4413ba" containerName="authorino"
Sep 08 15:08:30.291208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.291194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:30.293561 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.293543    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sd75n\""
Sep 08 15:08:30.302327 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.302298    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-140253605945136"]
Sep 08 15:08:30.350966 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.350936    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntp8n\" (UniqueName: \"kubernetes.io/projected/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34-kube-api-access-ntp8n\") pod \"test-curl-7752-140253605945136\" (UID: \"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34\") " pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:30.451704 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.451673    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntp8n\" (UniqueName: \"kubernetes.io/projected/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34-kube-api-access-ntp8n\") pod \"test-curl-7752-140253605945136\" (UID: \"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34\") " pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:30.465307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.465277    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntp8n\" (UniqueName: \"kubernetes.io/projected/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34-kube-api-access-ntp8n\") pod \"test-curl-7752-140253605945136\" (UID: \"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34\") " pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:30.600816 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.600745    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:30.872709 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.872677    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-140253605945136"]
Sep 08 15:08:30.875387 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:08:30.875362    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1f4212f_cf6c_447f_a399_4f1a7d4f1d34.slice/crio-4c8c864bb6a8c38843dc8503922d3596dbf72d89c0fb1594bedd975ee3ef0ca3 WatchSource:0}: Error finding container 4c8c864bb6a8c38843dc8503922d3596dbf72d89c0fb1594bedd975ee3ef0ca3: Status 404 returned error can't find the container with id 4c8c864bb6a8c38843dc8503922d3596dbf72d89c0fb1594bedd975ee3ef0ca3
Sep 08 15:08:30.959589 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:30.959562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-140253605945136" event={"ID":"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34","Type":"ContainerStarted","Data":"4c8c864bb6a8c38843dc8503922d3596dbf72d89c0fb1594bedd975ee3ef0ca3"}
Sep 08 15:08:31.935984 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:31.935943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:31.965934 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:31.965900    2578 generic.go:356] "Generic (PLEG): container finished" podID="b1f4212f-cf6c-447f-a399-4f1a7d4f1d34" containerID="51dfd3c74c98fc862227bbd8fcc702a4fcd065cc3a62627556a67d63c4b0f449" exitCode=0
Sep 08 15:08:31.966106 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:31.965997    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-140253605945136" event={"ID":"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34","Type":"ContainerDied","Data":"51dfd3c74c98fc862227bbd8fcc702a4fcd065cc3a62627556a67d63c4b0f449"}
Sep 08 15:08:32.091536 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:32.091479    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:32.991770 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:32.991549    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"]
Sep 08 15:08:32.996101 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:32.996077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:32.998130 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:32.998109    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a1592f-serving-cert\""
Sep 08 15:08:33.006876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.006842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"]
Sep 08 15:08:33.025311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.025282    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:33.073714 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.073688    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.073835 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.073744    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchpx\" (UniqueName: \"kubernetes.io/projected/7ca25c10-005a-4f12-bfc5-f61534d3f321-kube-api-access-rchpx\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.073835 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.073790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-metrics-certs\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.073920 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.073884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca25c10-005a-4f12-bfc5-f61534d3f321-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.123512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.123495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:33.127803 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.127773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:33.147355 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.147336    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7752-140253605945136_b1f4212f-cf6c-447f-a399-4f1a7d4f1d34/test-curl-7752-140253605945136/0.log"
Sep 08 15:08:33.173788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.173765    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-140253605945136"]
Sep 08 15:08:33.174227 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.174212    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntp8n\" (UniqueName: \"kubernetes.io/projected/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34-kube-api-access-ntp8n\") pod \"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34\" (UID: \"b1f4212f-cf6c-447f-a399-4f1a7d4f1d34\") "
Sep 08 15:08:33.174382 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.174366    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca25c10-005a-4f12-bfc5-f61534d3f321-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.174452 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.174404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.174452 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.174442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchpx\" (UniqueName: \"kubernetes.io/projected/7ca25c10-005a-4f12-bfc5-f61534d3f321-kube-api-access-rchpx\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.174609 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.174474    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-metrics-certs\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.174609 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:08:33.174560    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-serving-cert: secret "maas-api-e2e-inf-a-a1592f-serving-cert" not found
Sep 08 15:08:33.174734 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:08:33.174632    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls podName:7ca25c10-005a-4f12-bfc5-f61534d3f321 nodeName:}" failed. No retries permitted until 2026-09-08 15:08:33.674612811 +0000 UTC m=+792.326037098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls") pod "maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" (UID: "7ca25c10-005a-4f12-bfc5-f61534d3f321") : secret "maas-api-e2e-inf-a-a1592f-serving-cert" not found
Sep 08 15:08:33.175066 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.175046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca25c10-005a-4f12-bfc5-f61534d3f321-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.176898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.176875    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34-kube-api-access-ntp8n" (OuterVolumeSpecName: "kube-api-access-ntp8n") pod "b1f4212f-cf6c-447f-a399-4f1a7d4f1d34" (UID: "b1f4212f-cf6c-447f-a399-4f1a7d4f1d34"). InnerVolumeSpecName "kube-api-access-ntp8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:08:33.177198 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.177182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-metrics-certs\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.177897 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.177881    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-140253605945136"]
Sep 08 15:08:33.182974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.182889    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchpx\" (UniqueName: \"kubernetes.io/projected/7ca25c10-005a-4f12-bfc5-f61534d3f321-kube-api-access-rchpx\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.257372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.257338    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:33.275898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.275878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntp8n\" (UniqueName: \"kubernetes.io/projected/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34-kube-api-access-ntp8n\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:08:33.617672 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.617602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:33.678385 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.678351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.680802 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.680781    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls\") pod \"maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.714876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.714818    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:33.831842 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.831812    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1f4212f-cf6c-447f-a399-4f1a7d4f1d34" path="/var/lib/kubelet/pods/b1f4212f-cf6c-447f-a399-4f1a7d4f1d34/volumes"
Sep 08 15:08:33.909709 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.909642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:33.977882 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.977858    2578 scope.go:117] "RemoveContainer" containerID="51dfd3c74c98fc862227bbd8fcc702a4fcd065cc3a62627556a67d63c4b0f449"
Sep 08 15:08:33.978013 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:33.977882    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-140253605945136"
Sep 08 15:08:34.182411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:34.182389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"]
Sep 08 15:08:34.185040 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:08:34.185016    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca25c10_005a_4f12_bfc5_f61534d3f321.slice/crio-9d80661c66412afbafb64888a32a1885f6cf8e0f038ee7e95a9a35b8a5d0e418 WatchSource:0}: Error finding container 9d80661c66412afbafb64888a32a1885f6cf8e0f038ee7e95a9a35b8a5d0e418: Status 404 returned error can't find the container with id 9d80661c66412afbafb64888a32a1885f6cf8e0f038ee7e95a9a35b8a5d0e418
Sep 08 15:08:34.983708 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:34.983670    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" event={"ID":"7ca25c10-005a-4f12-bfc5-f61534d3f321","Type":"ContainerStarted","Data":"9d80661c66412afbafb64888a32a1885f6cf8e0f038ee7e95a9a35b8a5d0e418"}
Sep 08 15:08:36.074535 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:36.074488    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:36.175477 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:36.175443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:36.357247 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:36.357169    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:36.675390 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:36.675317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:37.129022 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:37.128988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:37.997257 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:37.997226    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" event={"ID":"7ca25c10-005a-4f12-bfc5-f61534d3f321","Type":"ContainerStarted","Data":"b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525"}
Sep 08 15:08:37.997410 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:37.997288    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:38.014998 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:38.014956    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" podStartSLOduration=3.144889741 podStartE2EDuration="6.01494123s" totalImagesPullingTime="2.870051489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:08:38.013613435 +0000 UTC m=+796.665037764" watchObservedRunningTime="2026-09-08 15:08:38.01494123 +0000 UTC m=+796.666365538"
Sep 08 15:08:38.633041 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:38.632988    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:38.938894 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:38.938826    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:39.357867 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:39.357831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:41.838384 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:41.838346    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:41.947637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:41.947612    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:42.135690 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.135622    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:42.211365 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.211334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:42.962048 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.962013    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"]
Sep 08 15:08:42.962517 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.962501    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1f4212f-cf6c-447f-a399-4f1a7d4f1d34" containerName="test-curl-7752-140253605945136"
Sep 08 15:08:42.962599 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.962587    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1f4212f-cf6c-447f-a399-4f1a7d4f1d34" containerName="test-curl-7752-140253605945136"
Sep 08 15:08:42.962708 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.962696    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1f4212f-cf6c-447f-a399-4f1a7d4f1d34" containerName="test-curl-7752-140253605945136"
Sep 08 15:08:42.966079 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.966059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:42.968253 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.968233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-6b2b2e-serving-cert\""
Sep 08 15:08:42.974233 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:42.974210    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"]
Sep 08 15:08:43.003692 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.003672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:43.065461 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.065440    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-maas-api-tls\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.065601 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.065468    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e857c-954d-4901-b298-a3279df49f70-openshift-service-ca\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.065601 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.065508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-metrics-certs\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.065720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.065648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4m2g\" (UniqueName: \"kubernetes.io/projected/b57e857c-954d-4901-b298-a3279df49f70-kube-api-access-v4m2g\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.113435 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.113415    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:43.166429 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.166406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-maas-api-tls\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.166513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.166432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e857c-954d-4901-b298-a3279df49f70-openshift-service-ca\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.166513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.166473    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-metrics-certs\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.166513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.166509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4m2g\" (UniqueName: \"kubernetes.io/projected/b57e857c-954d-4901-b298-a3279df49f70-kube-api-access-v4m2g\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.167161 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.167097    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e857c-954d-4901-b298-a3279df49f70-openshift-service-ca\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.168975 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.168948    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-maas-api-tls\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.169057 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.169013    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-metrics-certs\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.174444 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.174420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4m2g\" (UniqueName: \"kubernetes.io/projected/b57e857c-954d-4901-b298-a3279df49f70-kube-api-access-v4m2g\") pod \"maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.204343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.204317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:43.278422 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.278160    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:43.564965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.564942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"]
Sep 08 15:08:43.567132 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:08:43.567104    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57e857c_954d_4901_b298_a3279df49f70.slice/crio-917e1ce4ce22752b4102d18d7e53c0b691fef08f2c2dba25dfa5eaf3f33aee10 WatchSource:0}: Error finding container 917e1ce4ce22752b4102d18d7e53c0b691fef08f2c2dba25dfa5eaf3f33aee10: Status 404 returned error can't find the container with id 917e1ce4ce22752b4102d18d7e53c0b691fef08f2c2dba25dfa5eaf3f33aee10
Sep 08 15:08:43.692780 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:43.692748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:44.006246 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.006221    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:08:44.023098 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.023070    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn" event={"ID":"b57e857c-954d-4901-b298-a3279df49f70","Type":"ContainerStarted","Data":"917e1ce4ce22752b4102d18d7e53c0b691fef08f2c2dba25dfa5eaf3f33aee10"}
Sep 08 15:08:44.069851 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.069824    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"]
Sep 08 15:08:44.074899 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.074879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.077310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.077292    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4510c66e-serving-cert\""
Sep 08 15:08:44.081983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.081960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"]
Sep 08 15:08:44.176728 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.176706    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgjd9\" (UniqueName: \"kubernetes.io/projected/d0188476-f91f-4fec-a043-00c27daa4661-kube-api-access-zgjd9\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.176838 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.176737    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0188476-f91f-4fec-a043-00c27daa4661-openshift-service-ca\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.176838 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.176800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.176838 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.176817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-metrics-certs\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.278222 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.278161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgjd9\" (UniqueName: \"kubernetes.io/projected/d0188476-f91f-4fec-a043-00c27daa4661-kube-api-access-zgjd9\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.278222 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.278191    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0188476-f91f-4fec-a043-00c27daa4661-openshift-service-ca\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.278404 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.278242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.278404 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:08:44.278311    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-serving-cert: secret "maas-api-e2e-ait-4510c66e-serving-cert" not found
Sep 08 15:08:44.278404 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.278344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-metrics-certs\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.278404 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:08:44.278361    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls podName:d0188476-f91f-4fec-a043-00c27daa4661 nodeName:}" failed. No retries permitted until 2026-09-08 15:08:44.778344202 +0000 UTC m=+803.429768503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls") pod "maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" (UID: "d0188476-f91f-4fec-a043-00c27daa4661") : secret "maas-api-e2e-ait-4510c66e-serving-cert" not found
Sep 08 15:08:44.278938 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.278906    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0188476-f91f-4fec-a043-00c27daa4661-openshift-service-ca\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.280662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.280643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-metrics-certs\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.293933 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.293907    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgjd9\" (UniqueName: \"kubernetes.io/projected/d0188476-f91f-4fec-a043-00c27daa4661-kube-api-access-zgjd9\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.783280 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.783247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.785738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.785720    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls\") pod \"maas-api-e2e-ait-4510c66e-745b6d88ff-427kl\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:44.987694 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:44.987658    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:45.029855 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:45.029821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn" event={"ID":"b57e857c-954d-4901-b298-a3279df49f70","Type":"ContainerStarted","Data":"da4ab75a50791b95c170d797de318e1b4ec68d5400de0b2f7ecf892d36e83da7"}
Sep 08 15:08:45.030231 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:45.029910    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:45.050440 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:45.050393    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn" podStartSLOduration=2.50404196 podStartE2EDuration="3.050379281s" totalImagesPullingTime="546.337321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:08:45.049611443 +0000 UTC m=+803.701035751" watchObservedRunningTime="2026-09-08 15:08:45.050379281 +0000 UTC m=+803.701803590"
Sep 08 15:08:45.266888 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:45.266862    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"]
Sep 08 15:08:45.269716 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:08:45.269689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0188476_f91f_4fec_a043_00c27daa4661.slice/crio-fe5989e93549c972d61f2690f2bc2af5f80ca7708806d020efe729e03bc8ff90 WatchSource:0}: Error finding container fe5989e93549c972d61f2690f2bc2af5f80ca7708806d020efe729e03bc8ff90: Status 404 returned error can't find the container with id fe5989e93549c972d61f2690f2bc2af5f80ca7708806d020efe729e03bc8ff90
Sep 08 15:08:45.542259 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:45.542224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:46.036042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:46.036002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" event={"ID":"d0188476-f91f-4fec-a043-00c27daa4661","Type":"ContainerStarted","Data":"40111f3b1d85dc697f0902c48cf60b18209e69423df7f11eecd7ded9040e1353"}
Sep 08 15:08:46.036460 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:46.036055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" event={"ID":"d0188476-f91f-4fec-a043-00c27daa4661","Type":"ContainerStarted","Data":"fe5989e93549c972d61f2690f2bc2af5f80ca7708806d020efe729e03bc8ff90"}
Sep 08 15:08:46.036460 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:46.036298    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:46.057858 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:46.057809    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" podStartSLOduration=1.7387294880000002 podStartE2EDuration="2.05779361s" totalImagesPullingTime="319.064122ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:08:46.056326436 +0000 UTC m=+804.707750748" watchObservedRunningTime="2026-09-08 15:08:46.05779361 +0000 UTC m=+804.709217919"
Sep 08 15:08:47.404297 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:47.404266    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:47.505033 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:47.505000    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:49.322202 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:49.322165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:51.041673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:51.041645    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:08:51.234132 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:51.234100    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:51.336535 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:51.336442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:51.677429 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:51.677345    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:52.046098 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:52.046073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:08:54.072252 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.072191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:54.876356 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.876319    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"]
Sep 08 15:08:54.879886 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.879866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:54.882146 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.882123    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-be8647b6-kserve-self-signed-certs\""
Sep 08 15:08:54.890641 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.890616    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"]
Sep 08 15:08:54.968681 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.968656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-tmp-dir\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:54.968767 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.968694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82f4\" (UniqueName: \"kubernetes.io/projected/99777269-2a1b-49ae-b1bc-548af29974fd-kube-api-access-l82f4\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:54.968821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.968765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-model-cache\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:54.968821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.968800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-home\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:54.968906 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.968830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-dshm\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:54.968946 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:54.968902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99777269-2a1b-49ae-b1bc-548af29974fd-tls-certs\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.069989 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.069965    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-tmp-dir\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070127 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82f4\" (UniqueName: \"kubernetes.io/projected/99777269-2a1b-49ae-b1bc-548af29974fd-kube-api-access-l82f4\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070127 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-model-cache\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070127 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-home\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070127 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-dshm\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070127 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99777269-2a1b-49ae-b1bc-548af29974fd-tls-certs\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070401 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070360    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-tmp-dir\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070401 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-model-cache\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.070516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.070428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-home\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.072626 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.072603    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-dshm\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.073005 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.072633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99777269-2a1b-49ae-b1bc-548af29974fd-tls-certs\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.078324 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.078303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82f4\" (UniqueName: \"kubernetes.io/projected/99777269-2a1b-49ae-b1bc-548af29974fd-kube-api-access-l82f4\") pod \"e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") " pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.126549 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.126481    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:55.190268 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.190248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:08:55.462395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:55.462371    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"]
Sep 08 15:08:55.464384 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:08:55.464358    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99777269_2a1b_49ae_b1bc_548af29974fd.slice/crio-67e8e8c1227863a3edde8068da26fd7866e54bb567f87f13cc43d137fd9629c8 WatchSource:0}: Error finding container 67e8e8c1227863a3edde8068da26fd7866e54bb567f87f13cc43d137fd9629c8: Status 404 returned error can't find the container with id 67e8e8c1227863a3edde8068da26fd7866e54bb567f87f13cc43d137fd9629c8
Sep 08 15:08:56.087578 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.087542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc" event={"ID":"99777269-2a1b-49ae-b1bc-548af29974fd","Type":"ContainerStarted","Data":"cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1"}
Sep 08 15:08:56.087969 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.087585    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc" event={"ID":"99777269-2a1b-49ae-b1bc-548af29974fd","Type":"ContainerStarted","Data":"67e8e8c1227863a3edde8068da26fd7866e54bb567f87f13cc43d137fd9629c8"}
Sep 08 15:08:56.106018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.105973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc" podStartSLOduration=2.105956817 podStartE2EDuration="2.105956817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:08:56.104751973 +0000 UTC m=+814.756176283" watchObservedRunningTime="2026-09-08 15:08:56.105956817 +0000 UTC m=+814.757381126"
Sep 08 15:08:56.370498 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.370421    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:56.484949 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.484919    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:56.591200 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.591171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:56.805662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.805613    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:56.905320 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:56.905287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:57.227740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:57.227660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:57.568322 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:57.568288    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:08:58.455866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:08:58.455832    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:00.997848 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:00.997811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:01.094680 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:01.094653    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:02.174843 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:02.174804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:02.276214 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:02.276187    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:02.501267 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:02.501212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:04.662786 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:04.662756    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:04.763310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:04.763278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:05.190437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:05.190409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:09:05.190645 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:05.190452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:09:05.203964 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:05.203944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:09:05.378262 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:05.378234    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:06.112141 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:06.112105    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:06.141376 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:06.141354    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:09:06.819806 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:06.819778    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:06.900615 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:06.900578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:23.235724 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:23.235695    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:23.964262 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:23.964233    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"]
Sep 08 15:09:23.964499 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:23.964453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" podUID="d0188476-f91f-4fec-a043-00c27daa4661" containerName="maas-api" containerID="cri-o://40111f3b1d85dc697f0902c48cf60b18209e69423df7f11eecd7ded9040e1353" gracePeriod=30
Sep 08 15:09:24.188632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.188598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:24.212178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.212154    2578 generic.go:356] "Generic (PLEG): container finished" podID="d0188476-f91f-4fec-a043-00c27daa4661" containerID="40111f3b1d85dc697f0902c48cf60b18209e69423df7f11eecd7ded9040e1353" exitCode=0
Sep 08 15:09:24.212307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.212223    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" event={"ID":"d0188476-f91f-4fec-a043-00c27daa4661","Type":"ContainerDied","Data":"40111f3b1d85dc697f0902c48cf60b18209e69423df7f11eecd7ded9040e1353"}
Sep 08 15:09:24.212307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.212267    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl" event={"ID":"d0188476-f91f-4fec-a043-00c27daa4661","Type":"ContainerDied","Data":"fe5989e93549c972d61f2690f2bc2af5f80ca7708806d020efe729e03bc8ff90"}
Sep 08 15:09:24.212307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.212285    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe5989e93549c972d61f2690f2bc2af5f80ca7708806d020efe729e03bc8ff90"
Sep 08 15:09:24.217557 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.217495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:09:24.288010 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.287983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls\") pod \"d0188476-f91f-4fec-a043-00c27daa4661\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") "
Sep 08 15:09:24.288377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.288052    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0188476-f91f-4fec-a043-00c27daa4661-openshift-service-ca\") pod \"d0188476-f91f-4fec-a043-00c27daa4661\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") "
Sep 08 15:09:24.288377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.288107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgjd9\" (UniqueName: \"kubernetes.io/projected/d0188476-f91f-4fec-a043-00c27daa4661-kube-api-access-zgjd9\") pod \"d0188476-f91f-4fec-a043-00c27daa4661\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") "
Sep 08 15:09:24.288377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.288125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-metrics-certs\") pod \"d0188476-f91f-4fec-a043-00c27daa4661\" (UID: \"d0188476-f91f-4fec-a043-00c27daa4661\") "
Sep 08 15:09:24.288587 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.288449    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0188476-f91f-4fec-a043-00c27daa4661-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0188476-f91f-4fec-a043-00c27daa4661" (UID: "d0188476-f91f-4fec-a043-00c27daa4661"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:09:24.289962 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.289930    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:24.290688 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.290661    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0188476-f91f-4fec-a043-00c27daa4661-kube-api-access-zgjd9" (OuterVolumeSpecName: "kube-api-access-zgjd9") pod "d0188476-f91f-4fec-a043-00c27daa4661" (UID: "d0188476-f91f-4fec-a043-00c27daa4661"). InnerVolumeSpecName "kube-api-access-zgjd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:09:24.290772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.290711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d0188476-f91f-4fec-a043-00c27daa4661" (UID: "d0188476-f91f-4fec-a043-00c27daa4661"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:09:24.290772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.290727    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0188476-f91f-4fec-a043-00c27daa4661" (UID: "d0188476-f91f-4fec-a043-00c27daa4661"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:09:24.371786 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.371757    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:24.389009 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.388975    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0188476-f91f-4fec-a043-00c27daa4661-openshift-service-ca\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:09:24.389009 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.388997    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgjd9\" (UniqueName: \"kubernetes.io/projected/d0188476-f91f-4fec-a043-00c27daa4661-kube-api-access-zgjd9\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:09:24.389009 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.389007    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:09:24.389009 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.389016    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0188476-f91f-4fec-a043-00c27daa4661-maas-api-tls\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:09:24.860539 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.860497    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:24.959479 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:24.959451    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:25.097640 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.097617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:25.217408 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.217339    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"
Sep 08 15:09:25.258680 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.258657    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"]
Sep 08 15:09:25.267775 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.267753    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4510c66e-745b6d88ff-427kl"]
Sep 08 15:09:25.504276 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.504206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:25.832866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.832825    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0188476-f91f-4fec-a043-00c27daa4661" path="/var/lib/kubelet/pods/d0188476-f91f-4fec-a043-00c27daa4661/volumes"
Sep 08 15:09:25.863809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:25.863779    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:26.074089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:26.074062    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:28.974719 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:28.974686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:29.047840 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:29.047804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:29.207755 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:29.207725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:29.402394 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:29.402372    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:29.499276 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:29.499251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:35.165594 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:35.165561    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:45.175025 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.174990    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"]
Sep 08 15:09:45.175395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.175329    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0188476-f91f-4fec-a043-00c27daa4661" containerName="maas-api"
Sep 08 15:09:45.175395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.175342    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0188476-f91f-4fec-a043-00c27daa4661" containerName="maas-api"
Sep 08 15:09:45.175468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.175413    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0188476-f91f-4fec-a043-00c27daa4661" containerName="maas-api"
Sep 08 15:09:45.179678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.179662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.183758 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.183736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-be8647b6-kserve-self-signed-certs\""
Sep 08 15:09:45.188147 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.188123    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"]
Sep 08 15:09:45.251577 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.251551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-tmp-dir\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.251673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.251592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfkkm\" (UniqueName: \"kubernetes.io/projected/c77545d2-c445-45cb-8aa6-b5265ffd612c-kube-api-access-jfkkm\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.251673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.251634    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c77545d2-c445-45cb-8aa6-b5265ffd612c-tls-certs\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.251747 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.251694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-model-cache\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.251747 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.251713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-home\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.251747 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.251730    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-dshm\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c77545d2-c445-45cb-8aa6-b5265ffd612c-tls-certs\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352228    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-model-cache\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352350 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352337    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-home\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352388 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352361    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-dshm\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352421 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-tmp-dir\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352460 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352421    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfkkm\" (UniqueName: \"kubernetes.io/projected/c77545d2-c445-45cb-8aa6-b5265ffd612c-kube-api-access-jfkkm\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352597 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-model-cache\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352687 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-home\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.352804 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.352733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-tmp-dir\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.354756 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.354727    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-dshm\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.354880 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.354863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c77545d2-c445-45cb-8aa6-b5265ffd612c-tls-certs\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.361326 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.361304    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfkkm\" (UniqueName: \"kubernetes.io/projected/c77545d2-c445-45cb-8aa6-b5265ffd612c-kube-api-access-jfkkm\") pod \"e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") " pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.488612 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.488561    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:45.760813 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:45.760790    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"]
Sep 08 15:09:45.762900 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:09:45.762878    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc77545d2_c445_45cb_8aa6_b5265ffd612c.slice/crio-b7e692c619aab790d5d75934e3dfb7d3ca9e2ec33f6a3815108977e27a472daa WatchSource:0}: Error finding container b7e692c619aab790d5d75934e3dfb7d3ca9e2ec33f6a3815108977e27a472daa: Status 404 returned error can't find the container with id b7e692c619aab790d5d75934e3dfb7d3ca9e2ec33f6a3815108977e27a472daa
Sep 08 15:09:46.008166 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:46.008133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:46.311713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:46.311665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48" event={"ID":"c77545d2-c445-45cb-8aa6-b5265ffd612c","Type":"ContainerStarted","Data":"83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d"}
Sep 08 15:09:46.312104 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:46.311726    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48" event={"ID":"c77545d2-c445-45cb-8aa6-b5265ffd612c","Type":"ContainerStarted","Data":"b7e692c619aab790d5d75934e3dfb7d3ca9e2ec33f6a3815108977e27a472daa"}
Sep 08 15:09:46.328825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:46.328762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48" podStartSLOduration=1.328748548 podStartE2EDuration="1.328748548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:09:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:09:46.327416797 +0000 UTC m=+864.978841105" watchObservedRunningTime="2026-09-08 15:09:46.328748548 +0000 UTC m=+864.980172857"
Sep 08 15:09:55.489412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:55.489377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:55.489412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:55.489419    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:55.501971 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:55.501949    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:56.375827 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:56.375799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:09:59.538768 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:59.538727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:59.643898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:59.643863    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:59.739022 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:59.738996    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:59.806761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:59.806704    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:09:59.926247 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:09:59.926212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:00.377546 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:00.377496    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:00.697932 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:00.697855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:01.482794 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:01.482760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:03.623581 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:03.623550    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:03.723211 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:03.723181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:04.783125 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:04.783095    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:04.882546 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:04.879316    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:04.999277 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:04.999245    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:05.361677 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.361645    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:05.462882 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.462854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:05.489313 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.489281    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"]
Sep 08 15:10:05.492923 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.492900    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.495536 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.495501    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-cc8fbdcf\""
Sep 08 15:10:05.495663 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.495506    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-cc8fbdcf-dockercfg-7fnx6\""
Sep 08 15:10:05.506507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.506482    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"]
Sep 08 15:10:05.614165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.614108    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmftk\" (UniqueName: \"kubernetes.io/projected/30d86582-5acd-485c-8e43-7658b210bc85-kube-api-access-dmftk\") pod \"payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") " pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.614165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.614139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d86582-5acd-485c-8e43-7658b210bc85-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") " pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.714986 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.714955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmftk\" (UniqueName: \"kubernetes.io/projected/30d86582-5acd-485c-8e43-7658b210bc85-kube-api-access-dmftk\") pod \"payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") " pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.715154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.714997    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d86582-5acd-485c-8e43-7658b210bc85-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") " pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.715706 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.715687    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d86582-5acd-485c-8e43-7658b210bc85-plugins-config-volume\") pod \"payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") " pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.722713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.722690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmftk\" (UniqueName: \"kubernetes.io/projected/30d86582-5acd-485c-8e43-7658b210bc85-kube-api-access-dmftk\") pod \"payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") " pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:05.807279 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:05.807255    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:06.082656 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:06.082631    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"]
Sep 08 15:10:06.085251 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:10:06.085217    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d86582_5acd_485c_8e43_7658b210bc85.slice/crio-1d1133e1416186d0833abfd6ca9e768b6a6d6cd39f81a1408be8f156ad381d81 WatchSource:0}: Error finding container 1d1133e1416186d0833abfd6ca9e768b6a6d6cd39f81a1408be8f156ad381d81: Status 404 returned error can't find the container with id 1d1133e1416186d0833abfd6ca9e768b6a6d6cd39f81a1408be8f156ad381d81
Sep 08 15:10:06.410767 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:06.410689    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6" event={"ID":"30d86582-5acd-485c-8e43-7658b210bc85","Type":"ContainerStarted","Data":"1d1133e1416186d0833abfd6ca9e768b6a6d6cd39f81a1408be8f156ad381d81"}
Sep 08 15:10:10.450434 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:10.450392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6" event={"ID":"30d86582-5acd-485c-8e43-7658b210bc85","Type":"ContainerStarted","Data":"46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46"}
Sep 08 15:10:10.451064 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:10.450665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:10.470781 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:10.470709    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6" podStartSLOduration=1.218761503 podStartE2EDuration="5.47068681s" totalImagesPullingTime="4.251925307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:10:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:10:10.467769613 +0000 UTC m=+889.119193929" watchObservedRunningTime="2026-09-08 15:10:10.47068681 +0000 UTC m=+889.122111126"
Sep 08 15:10:12.467858 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.467822    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"]
Sep 08 15:10:12.471219 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.471199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.473670 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.473647    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6b2b2e\"/\"openshift-service-ca.crt\""
Sep 08 15:10:12.474418 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.474397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6b2b2e\"/\"default-dockercfg-928bz\""
Sep 08 15:10:12.474418 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.474405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-6b2b2e\"/\"autores-model-6b2b2e-kserve-self-signed-certs\""
Sep 08 15:10:12.474418 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.474415    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-6b2b2e\"/\"kube-root-ca.crt\""
Sep 08 15:10:12.479804 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.479782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"]
Sep 08 15:10:12.576821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.576798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-model-cache\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.576921 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.576848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzhpc\" (UniqueName: \"kubernetes.io/projected/06c179e2-fb3b-4618-a137-5e8218d38578-kube-api-access-qzhpc\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.576921 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.576902    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-home\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.576992 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.576925    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-dshm\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.576992 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.576974    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c179e2-fb3b-4618-a137-5e8218d38578-tls-certs\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.577099 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.577003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-tmp-dir\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678024 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.677993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-model-cache\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzhpc\" (UniqueName: \"kubernetes.io/projected/06c179e2-fb3b-4618-a137-5e8218d38578-kube-api-access-qzhpc\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-home\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678150    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-dshm\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c179e2-fb3b-4618-a137-5e8218d38578-tls-certs\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678207    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-tmp-dir\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678452 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-model-cache\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678490    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-home\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.678720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.678700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-tmp-dir\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.680750 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.680730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-dshm\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.681012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.680989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c179e2-fb3b-4618-a137-5e8218d38578-tls-certs\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.686796 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.686775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzhpc\" (UniqueName: \"kubernetes.io/projected/06c179e2-fb3b-4618-a137-5e8218d38578-kube-api-access-qzhpc\") pod \"autores-model-6b2b2e-kserve-5d97d674bc-znl7p\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") " pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.782015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.781987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:12.923796 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:12.923768    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:13.057014 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:13.056993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"]
Sep 08 15:10:13.059474 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:10:13.059448    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06c179e2_fb3b_4618_a137_5e8218d38578.slice/crio-dad0903d6f4463010dfcd963891e257a82f4e1ede03355eec8706ff27d4c2f67 WatchSource:0}: Error finding container dad0903d6f4463010dfcd963891e257a82f4e1ede03355eec8706ff27d4c2f67: Status 404 returned error can't find the container with id dad0903d6f4463010dfcd963891e257a82f4e1ede03355eec8706ff27d4c2f67
Sep 08 15:10:13.467466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:13.467346    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p" event={"ID":"06c179e2-fb3b-4618-a137-5e8218d38578","Type":"ContainerStarted","Data":"04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522"}
Sep 08 15:10:13.467466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:13.467392    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p" event={"ID":"06c179e2-fb3b-4618-a137-5e8218d38578","Type":"ContainerStarted","Data":"dad0903d6f4463010dfcd963891e257a82f4e1ede03355eec8706ff27d4c2f67"}
Sep 08 15:10:13.485772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:13.485728    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p" podStartSLOduration=1.485713932 podStartE2EDuration="1.485713932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:10:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:10:13.484627392 +0000 UTC m=+892.136051704" watchObservedRunningTime="2026-09-08 15:10:13.485713932 +0000 UTC m=+892.137138241"
Sep 08 15:10:21.457620 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:21.457589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:22.782690 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:22.782643    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:22.782690 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:22.782701    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:22.795132 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:22.795113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:23.526507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:23.526481    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:10:26.083220 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:26.083191    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:35.458757 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.458725    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:35.485074 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.485045    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"]
Sep 08 15:10:35.485426 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.485367    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6" podUID="30d86582-5acd-485c-8e43-7658b210bc85" containerName="payload-processing" containerID="cri-o://46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46" gracePeriod=30
Sep 08 15:10:35.555319 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.555283    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:35.658729 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.658650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:35.730151 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.730127    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:35.750334 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.750307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d86582-5acd-485c-8e43-7658b210bc85-plugins-config-volume\") pod \"30d86582-5acd-485c-8e43-7658b210bc85\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") "
Sep 08 15:10:35.750459 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.750372    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmftk\" (UniqueName: \"kubernetes.io/projected/30d86582-5acd-485c-8e43-7658b210bc85-kube-api-access-dmftk\") pod \"30d86582-5acd-485c-8e43-7658b210bc85\" (UID: \"30d86582-5acd-485c-8e43-7658b210bc85\") "
Sep 08 15:10:35.750718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.750690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d86582-5acd-485c-8e43-7658b210bc85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30d86582-5acd-485c-8e43-7658b210bc85" (UID: "30d86582-5acd-485c-8e43-7658b210bc85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:10:35.752845 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.752777    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:35.753110 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.753082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d86582-5acd-485c-8e43-7658b210bc85-kube-api-access-dmftk" (OuterVolumeSpecName: "kube-api-access-dmftk") pod "30d86582-5acd-485c-8e43-7658b210bc85" (UID: "30d86582-5acd-485c-8e43-7658b210bc85"). InnerVolumeSpecName "kube-api-access-dmftk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:10:35.852062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.852040    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d86582-5acd-485c-8e43-7658b210bc85-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:10:35.852147 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:35.852061    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmftk\" (UniqueName: \"kubernetes.io/projected/30d86582-5acd-485c-8e43-7658b210bc85-kube-api-access-dmftk\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:10:36.574334 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.574302    2578 generic.go:356] "Generic (PLEG): container finished" podID="30d86582-5acd-485c-8e43-7658b210bc85" containerID="46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46" exitCode=0
Sep 08 15:10:36.574738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.574347    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"
Sep 08 15:10:36.574738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.574391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6" event={"ID":"30d86582-5acd-485c-8e43-7658b210bc85","Type":"ContainerDied","Data":"46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46"}
Sep 08 15:10:36.574738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.574440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6" event={"ID":"30d86582-5acd-485c-8e43-7658b210bc85","Type":"ContainerDied","Data":"1d1133e1416186d0833abfd6ca9e768b6a6d6cd39f81a1408be8f156ad381d81"}
Sep 08 15:10:36.574738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.574462    2578 scope.go:117] "RemoveContainer" containerID="46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46"
Sep 08 15:10:36.583068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.583051    2578 scope.go:117] "RemoveContainer" containerID="46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46"
Sep 08 15:10:36.583281 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:10:36.583262    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46\": container with ID starting with 46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46 not found: ID does not exist" containerID="46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46"
Sep 08 15:10:36.583344 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.583293    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46"} err="failed to get container status \"46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46\": rpc error: code = NotFound desc = could not find container \"46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46\": container with ID starting with 46b1f886ab051e42b8e00bcbee3cbac5aa94abd4f6a578fb5cf1db7ceff1fc46 not found: ID does not exist"
Sep 08 15:10:36.590269 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.590250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"]
Sep 08 15:10:36.593745 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.593727    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-cc8fbdcf-7d4545b49-jrrj6"]
Sep 08 15:10:36.822294 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:36.822265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:37.261996 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:37.261962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:37.600013 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:37.599932    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:37.768178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:37.768146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:37.832257 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:37.832229    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30d86582-5acd-485c-8e43-7658b210bc85" path="/var/lib/kubelet/pods/30d86582-5acd-485c-8e43-7658b210bc85/volumes"
Sep 08 15:10:39.915849 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:39.915811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:41.659046 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.659015    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"]
Sep 08 15:10:41.659412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.659363    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30d86582-5acd-485c-8e43-7658b210bc85" containerName="payload-processing"
Sep 08 15:10:41.659412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.659375    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="30d86582-5acd-485c-8e43-7658b210bc85" containerName="payload-processing"
Sep 08 15:10:41.659484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.659427    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="30d86582-5acd-485c-8e43-7658b210bc85" containerName="payload-processing"
Sep 08 15:10:41.663922 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.663898    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.667672 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.667653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9b84af36\""
Sep 08 15:10:41.667797 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.667730    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9b84af36-dockercfg-z8v4m\""
Sep 08 15:10:41.673894 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.673872    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"]
Sep 08 15:10:41.695236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.695214    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"]
Sep 08 15:10:41.696770 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.696753    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg4ck\" (UniqueName: \"kubernetes.io/projected/d90f72c2-c416-47c0-98f6-bc0582de7405-kube-api-access-sg4ck\") pod \"payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.696902 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.696882    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90f72c2-c416-47c0-98f6-bc0582de7405-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.697472 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.697461    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.708392 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.708372    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"]
Sep 08 15:10:41.797495 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.797462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblp8\" (UniqueName: \"kubernetes.io/projected/60793971-d1f1-4c97-a30d-1a5bf6a3f162-kube-api-access-nblp8\") pod \"payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") " pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.797643 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.797546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90f72c2-c416-47c0-98f6-bc0582de7405-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.797643 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.797580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg4ck\" (UniqueName: \"kubernetes.io/projected/d90f72c2-c416-47c0-98f6-bc0582de7405-kube-api-access-sg4ck\") pod \"payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.797643 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.797631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60793971-d1f1-4c97-a30d-1a5bf6a3f162-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") " pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.798165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.798146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90f72c2-c416-47c0-98f6-bc0582de7405-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.805508 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.805488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg4ck\" (UniqueName: \"kubernetes.io/projected/d90f72c2-c416-47c0-98f6-bc0582de7405-kube-api-access-sg4ck\") pod \"payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:41.898314 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.898290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblp8\" (UniqueName: \"kubernetes.io/projected/60793971-d1f1-4c97-a30d-1a5bf6a3f162-kube-api-access-nblp8\") pod \"payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") " pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.898440 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.898365    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60793971-d1f1-4c97-a30d-1a5bf6a3f162-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") " pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.899048 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.899031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60793971-d1f1-4c97-a30d-1a5bf6a3f162-plugins-config-volume\") pod \"payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") " pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.906014 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.905992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblp8\" (UniqueName: \"kubernetes.io/projected/60793971-d1f1-4c97-a30d-1a5bf6a3f162-kube-api-access-nblp8\") pod \"payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") " pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:41.975464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:41.975408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:42.007540 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.007495    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:42.249879 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.249851    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"]
Sep 08 15:10:42.251925 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:10:42.251893    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90f72c2_c416_47c0_98f6_bc0582de7405.slice/crio-9be26e2fed730b9baf14a1937fa2eb661ded4a1fadf1d4eec69f6fb45998cb29 WatchSource:0}: Error finding container 9be26e2fed730b9baf14a1937fa2eb661ded4a1fadf1d4eec69f6fb45998cb29: Status 404 returned error can't find the container with id 9be26e2fed730b9baf14a1937fa2eb661ded4a1fadf1d4eec69f6fb45998cb29
Sep 08 15:10:42.298013 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.297992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"]
Sep 08 15:10:42.299985 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:10:42.299961    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60793971_d1f1_4c97_a30d_1a5bf6a3f162.slice/crio-3e70f17e77ca507de56bcf1b89cd8ec5553237ece3a5350c247cd0bd1ce6fc05 WatchSource:0}: Error finding container 3e70f17e77ca507de56bcf1b89cd8ec5553237ece3a5350c247cd0bd1ce6fc05: Status 404 returned error can't find the container with id 3e70f17e77ca507de56bcf1b89cd8ec5553237ece3a5350c247cd0bd1ce6fc05
Sep 08 15:10:42.607966 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.607928    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk" event={"ID":"d90f72c2-c416-47c0-98f6-bc0582de7405","Type":"ContainerStarted","Data":"498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb"}
Sep 08 15:10:42.608148 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.607999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk" event={"ID":"d90f72c2-c416-47c0-98f6-bc0582de7405","Type":"ContainerStarted","Data":"9be26e2fed730b9baf14a1937fa2eb661ded4a1fadf1d4eec69f6fb45998cb29"}
Sep 08 15:10:42.608226 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.608179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:42.609847 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.609817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2" event={"ID":"60793971-d1f1-4c97-a30d-1a5bf6a3f162","Type":"ContainerStarted","Data":"e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e"}
Sep 08 15:10:42.609979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.609869    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2" event={"ID":"60793971-d1f1-4c97-a30d-1a5bf6a3f162","Type":"ContainerStarted","Data":"3e70f17e77ca507de56bcf1b89cd8ec5553237ece3a5350c247cd0bd1ce6fc05"}
Sep 08 15:10:42.609979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.609908    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:42.624553 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.624492    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk" podStartSLOduration=1.6244743179999999 podStartE2EDuration="1.624474318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:10:42.623569932 +0000 UTC m=+921.274994242" watchObservedRunningTime="2026-09-08 15:10:42.624474318 +0000 UTC m=+921.275898628"
Sep 08 15:10:42.641497 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:42.641456    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2" podStartSLOduration=1.641444318 podStartE2EDuration="1.641444318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:10:42.639352815 +0000 UTC m=+921.290777125" watchObservedRunningTime="2026-09-08 15:10:42.641444318 +0000 UTC m=+921.292868627"
Sep 08 15:10:43.930150 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:43.930121    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:44.916953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:44.916923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:45.872848 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:45.872809    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:49.395418 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:49.395382    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"]
Sep 08 15:10:49.426316 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:49.426287    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:50.372388 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:50.372356    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"]
Sep 08 15:10:50.397688 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:50.397662    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:51.975002 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:51.974974    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"]
Sep 08 15:10:51.999350 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:51.999319    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:53.617669 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:53.617639    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:10:53.618014 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:53.617711    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:10:55.425730 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.425699    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"]
Sep 08 15:10:55.426123 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.425937    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn" podUID="b57e857c-954d-4901-b298-a3279df49f70" containerName="maas-api" containerID="cri-o://da4ab75a50791b95c170d797de318e1b4ec68d5400de0b2f7ecf892d36e83da7" gracePeriod=30
Sep 08 15:10:55.460437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.460411    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:55.559844 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.559811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:55.657564 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.657536    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:55.675660 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.675634    2578 generic.go:356] "Generic (PLEG): container finished" podID="b57e857c-954d-4901-b298-a3279df49f70" containerID="da4ab75a50791b95c170d797de318e1b4ec68d5400de0b2f7ecf892d36e83da7" exitCode=0
Sep 08 15:10:55.675805 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.675705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn" event={"ID":"b57e857c-954d-4901-b298-a3279df49f70","Type":"ContainerDied","Data":"da4ab75a50791b95c170d797de318e1b4ec68d5400de0b2f7ecf892d36e83da7"}
Sep 08 15:10:55.675805 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.675753    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn" event={"ID":"b57e857c-954d-4901-b298-a3279df49f70","Type":"ContainerDied","Data":"917e1ce4ce22752b4102d18d7e53c0b691fef08f2c2dba25dfa5eaf3f33aee10"}
Sep 08 15:10:55.675805 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.675773    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="917e1ce4ce22752b4102d18d7e53c0b691fef08f2c2dba25dfa5eaf3f33aee10"
Sep 08 15:10:55.683363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.683343    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:10:55.722401 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.722370    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:55.816015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.815989    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4m2g\" (UniqueName: \"kubernetes.io/projected/b57e857c-954d-4901-b298-a3279df49f70-kube-api-access-v4m2g\") pod \"b57e857c-954d-4901-b298-a3279df49f70\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") "
Sep 08 15:10:55.816165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.816077    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e857c-954d-4901-b298-a3279df49f70-openshift-service-ca\") pod \"b57e857c-954d-4901-b298-a3279df49f70\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") "
Sep 08 15:10:55.816165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.816125    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-maas-api-tls\") pod \"b57e857c-954d-4901-b298-a3279df49f70\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") "
Sep 08 15:10:55.816305 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.816189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-metrics-certs\") pod \"b57e857c-954d-4901-b298-a3279df49f70\" (UID: \"b57e857c-954d-4901-b298-a3279df49f70\") "
Sep 08 15:10:55.816406 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.816375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57e857c-954d-4901-b298-a3279df49f70-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b57e857c-954d-4901-b298-a3279df49f70" (UID: "b57e857c-954d-4901-b298-a3279df49f70"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:10:55.816560 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.816480    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b57e857c-954d-4901-b298-a3279df49f70-openshift-service-ca\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:10:55.818593 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.818570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b57e857c-954d-4901-b298-a3279df49f70" (UID: "b57e857c-954d-4901-b298-a3279df49f70"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:10:55.818958 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.818933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b57e857c-954d-4901-b298-a3279df49f70" (UID: "b57e857c-954d-4901-b298-a3279df49f70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:10:55.818958 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.818947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57e857c-954d-4901-b298-a3279df49f70-kube-api-access-v4m2g" (OuterVolumeSpecName: "kube-api-access-v4m2g") pod "b57e857c-954d-4901-b298-a3279df49f70" (UID: "b57e857c-954d-4901-b298-a3279df49f70"). InnerVolumeSpecName "kube-api-access-v4m2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:10:55.917624 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.917602    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4m2g\" (UniqueName: \"kubernetes.io/projected/b57e857c-954d-4901-b298-a3279df49f70-kube-api-access-v4m2g\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:10:55.917624 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.917623    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-maas-api-tls\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:10:55.917748 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:55.917633    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e857c-954d-4901-b298-a3279df49f70-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:10:56.027967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:56.027936    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:56.680725 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:56.680696    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"
Sep 08 15:10:56.698089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:56.698062    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"]
Sep 08 15:10:56.704280 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:56.704257    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-6b2b2e-dd4454895-vn7wn"]
Sep 08 15:10:57.185784 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:57.185754    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:10:57.831598 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:10:57.831565    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57e857c-954d-4901-b298-a3279df49f70" path="/var/lib/kubelet/pods/b57e857c-954d-4901-b298-a3279df49f70/volumes"
Sep 08 15:11:01.107420 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:01.107382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:01.934333 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:01.934302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:04.416076 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.416028    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc" podUID="99777269-2a1b-49ae-b1bc-548af29974fd" containerName="main" containerID="cri-o://cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1" gracePeriod=45
Sep 08 15:11:04.662979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.662959    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:11:04.720067 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.720005    2578 generic.go:356] "Generic (PLEG): container finished" podID="99777269-2a1b-49ae-b1bc-548af29974fd" containerID="cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1" exitCode=0
Sep 08 15:11:04.720067 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.720039    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"
Sep 08 15:11:04.720067 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.720055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc" event={"ID":"99777269-2a1b-49ae-b1bc-548af29974fd","Type":"ContainerDied","Data":"cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1"}
Sep 08 15:11:04.720242 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.720092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc" event={"ID":"99777269-2a1b-49ae-b1bc-548af29974fd","Type":"ContainerDied","Data":"67e8e8c1227863a3edde8068da26fd7866e54bb567f87f13cc43d137fd9629c8"}
Sep 08 15:11:04.720242 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.720116    2578 scope.go:117] "RemoveContainer" containerID="cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1"
Sep 08 15:11:04.728652 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.728633    2578 scope.go:117] "RemoveContainer" containerID="cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1"
Sep 08 15:11:04.728925 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:11:04.728909    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1\": container with ID starting with cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1 not found: ID does not exist" containerID="cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1"
Sep 08 15:11:04.728977 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.728932    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1"} err="failed to get container status \"cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1\": rpc error: code = NotFound desc = could not find container \"cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1\": container with ID starting with cf3a0aeaa72d08c0153db8fce2f5064c6be1675a8bdc3c9b6b6a07ea8bdab9e1 not found: ID does not exist"
Sep 08 15:11:04.788061 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788031    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-tmp-dir\") pod \"99777269-2a1b-49ae-b1bc-548af29974fd\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") "
Sep 08 15:11:04.788203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788118    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l82f4\" (UniqueName: \"kubernetes.io/projected/99777269-2a1b-49ae-b1bc-548af29974fd-kube-api-access-l82f4\") pod \"99777269-2a1b-49ae-b1bc-548af29974fd\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") "
Sep 08 15:11:04.788203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788153    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-dshm\") pod \"99777269-2a1b-49ae-b1bc-548af29974fd\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") "
Sep 08 15:11:04.788203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788192    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99777269-2a1b-49ae-b1bc-548af29974fd-tls-certs\") pod \"99777269-2a1b-49ae-b1bc-548af29974fd\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") "
Sep 08 15:11:04.788370 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788229    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-home\") pod \"99777269-2a1b-49ae-b1bc-548af29974fd\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") "
Sep 08 15:11:04.788425 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788385    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-model-cache\") pod \"99777269-2a1b-49ae-b1bc-548af29974fd\" (UID: \"99777269-2a1b-49ae-b1bc-548af29974fd\") "
Sep 08 15:11:04.788696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788667    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-home" (OuterVolumeSpecName: "home") pod "99777269-2a1b-49ae-b1bc-548af29974fd" (UID: "99777269-2a1b-49ae-b1bc-548af29974fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:04.788815 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99777269-2a1b-49ae-b1bc-548af29974fd" (UID: "99777269-2a1b-49ae-b1bc-548af29974fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:04.788896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788818    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:04.788896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.788860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-model-cache" (OuterVolumeSpecName: "model-cache") pod "99777269-2a1b-49ae-b1bc-548af29974fd" (UID: "99777269-2a1b-49ae-b1bc-548af29974fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:04.790857 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.790831    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-dshm" (OuterVolumeSpecName: "dshm") pod "99777269-2a1b-49ae-b1bc-548af29974fd" (UID: "99777269-2a1b-49ae-b1bc-548af29974fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:04.791115 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.791098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99777269-2a1b-49ae-b1bc-548af29974fd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99777269-2a1b-49ae-b1bc-548af29974fd" (UID: "99777269-2a1b-49ae-b1bc-548af29974fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:11:04.791178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.791121    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99777269-2a1b-49ae-b1bc-548af29974fd-kube-api-access-l82f4" (OuterVolumeSpecName: "kube-api-access-l82f4") pod "99777269-2a1b-49ae-b1bc-548af29974fd" (UID: "99777269-2a1b-49ae-b1bc-548af29974fd"). InnerVolumeSpecName "kube-api-access-l82f4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:11:04.890022 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.889997    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99777269-2a1b-49ae-b1bc-548af29974fd-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:04.890022 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.890023    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:04.890192 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.890034    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:04.890192 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.890045    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l82f4\" (UniqueName: \"kubernetes.io/projected/99777269-2a1b-49ae-b1bc-548af29974fd-kube-api-access-l82f4\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:04.890192 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:04.890056    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99777269-2a1b-49ae-b1bc-548af29974fd-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:05.040719 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.040696    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"]
Sep 08 15:11:05.046292 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.046272    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-be8647b6-kserve-fc6cc78bc-59ppc"]
Sep 08 15:11:05.393700 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.393593    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48" podUID="c77545d2-c445-45cb-8aa6-b5265ffd612c" containerName="main" containerID="cri-o://83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d" gracePeriod=45
Sep 08 15:11:05.650403 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.650357    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:11:05.732935 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.732904    2578 generic.go:356] "Generic (PLEG): container finished" podID="c77545d2-c445-45cb-8aa6-b5265ffd612c" containerID="83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d" exitCode=0
Sep 08 15:11:05.733092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.732974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"
Sep 08 15:11:05.733092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.732981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48" event={"ID":"c77545d2-c445-45cb-8aa6-b5265ffd612c","Type":"ContainerDied","Data":"83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d"}
Sep 08 15:11:05.733092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.733021    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48" event={"ID":"c77545d2-c445-45cb-8aa6-b5265ffd612c","Type":"ContainerDied","Data":"b7e692c619aab790d5d75934e3dfb7d3ca9e2ec33f6a3815108977e27a472daa"}
Sep 08 15:11:05.733092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.733038    2578 scope.go:117] "RemoveContainer" containerID="83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d"
Sep 08 15:11:05.741460 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.741443    2578 scope.go:117] "RemoveContainer" containerID="83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d"
Sep 08 15:11:05.741762 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:11:05.741743    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d\": container with ID starting with 83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d not found: ID does not exist" containerID="83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d"
Sep 08 15:11:05.741808 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.741775    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d"} err="failed to get container status \"83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d\": rpc error: code = NotFound desc = could not find container \"83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d\": container with ID starting with 83a14ce6fdcb128a8053e06b6e113265fcee9f032bd136c4e52813e0e91a3c2d not found: ID does not exist"
Sep 08 15:11:05.796601 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796578    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-model-cache\") pod \"c77545d2-c445-45cb-8aa6-b5265ffd612c\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") "
Sep 08 15:11:05.796697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-dshm\") pod \"c77545d2-c445-45cb-8aa6-b5265ffd612c\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") "
Sep 08 15:11:05.796697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-home\") pod \"c77545d2-c445-45cb-8aa6-b5265ffd612c\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") "
Sep 08 15:11:05.796697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796687    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c77545d2-c445-45cb-8aa6-b5265ffd612c-tls-certs\") pod \"c77545d2-c445-45cb-8aa6-b5265ffd612c\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") "
Sep 08 15:11:05.796846 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796722    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-tmp-dir\") pod \"c77545d2-c445-45cb-8aa6-b5265ffd612c\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") "
Sep 08 15:11:05.796846 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796743    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfkkm\" (UniqueName: \"kubernetes.io/projected/c77545d2-c445-45cb-8aa6-b5265ffd612c-kube-api-access-jfkkm\") pod \"c77545d2-c445-45cb-8aa6-b5265ffd612c\" (UID: \"c77545d2-c445-45cb-8aa6-b5265ffd612c\") "
Sep 08 15:11:05.797007 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796862    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-model-cache" (OuterVolumeSpecName: "model-cache") pod "c77545d2-c445-45cb-8aa6-b5265ffd612c" (UID: "c77545d2-c445-45cb-8aa6-b5265ffd612c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:05.797007 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.796947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-home" (OuterVolumeSpecName: "home") pod "c77545d2-c445-45cb-8aa6-b5265ffd612c" (UID: "c77545d2-c445-45cb-8aa6-b5265ffd612c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:05.797120 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.797014    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:05.797120 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.797039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c77545d2-c445-45cb-8aa6-b5265ffd612c" (UID: "c77545d2-c445-45cb-8aa6-b5265ffd612c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:05.798736 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.798715    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-dshm" (OuterVolumeSpecName: "dshm") pod "c77545d2-c445-45cb-8aa6-b5265ffd612c" (UID: "c77545d2-c445-45cb-8aa6-b5265ffd612c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:05.798892 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.798876    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77545d2-c445-45cb-8aa6-b5265ffd612c-kube-api-access-jfkkm" (OuterVolumeSpecName: "kube-api-access-jfkkm") pod "c77545d2-c445-45cb-8aa6-b5265ffd612c" (UID: "c77545d2-c445-45cb-8aa6-b5265ffd612c"). InnerVolumeSpecName "kube-api-access-jfkkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:11:05.799059 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.799029    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77545d2-c445-45cb-8aa6-b5265ffd612c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c77545d2-c445-45cb-8aa6-b5265ffd612c" (UID: "c77545d2-c445-45cb-8aa6-b5265ffd612c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:11:05.832131 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.832106    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99777269-2a1b-49ae-b1bc-548af29974fd" path="/var/lib/kubelet/pods/99777269-2a1b-49ae-b1bc-548af29974fd/volumes"
Sep 08 15:11:05.898135 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.898113    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:05.898135 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.898134    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:05.898255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.898143    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c77545d2-c445-45cb-8aa6-b5265ffd612c-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:05.898255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.898152    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77545d2-c445-45cb-8aa6-b5265ffd612c-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:05.898255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:05.898160    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfkkm\" (UniqueName: \"kubernetes.io/projected/c77545d2-c445-45cb-8aa6-b5265ffd612c-kube-api-access-jfkkm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:06.047885 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:06.047861    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"]
Sep 08 15:11:06.051721 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:06.051695    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-be8647b6-kserve-6d979c485-fzd48"]
Sep 08 15:11:06.994330 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:06.994274    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p" podUID="06c179e2-fb3b-4618-a137-5e8218d38578" containerName="main" containerID="cri-o://04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522" gracePeriod=45
Sep 08 15:11:07.244498 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.244442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:11:07.410597 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-home\") pod \"06c179e2-fb3b-4618-a137-5e8218d38578\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") "
Sep 08 15:11:07.410749 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410639    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-model-cache\") pod \"06c179e2-fb3b-4618-a137-5e8218d38578\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") "
Sep 08 15:11:07.410749 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410668    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzhpc\" (UniqueName: \"kubernetes.io/projected/06c179e2-fb3b-4618-a137-5e8218d38578-kube-api-access-qzhpc\") pod \"06c179e2-fb3b-4618-a137-5e8218d38578\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") "
Sep 08 15:11:07.410749 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-dshm\") pod \"06c179e2-fb3b-4618-a137-5e8218d38578\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") "
Sep 08 15:11:07.410881 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410779    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-tmp-dir\") pod \"06c179e2-fb3b-4618-a137-5e8218d38578\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") "
Sep 08 15:11:07.410881 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410815    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c179e2-fb3b-4618-a137-5e8218d38578-tls-certs\") pod \"06c179e2-fb3b-4618-a137-5e8218d38578\" (UID: \"06c179e2-fb3b-4618-a137-5e8218d38578\") "
Sep 08 15:11:07.410881 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410812    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-home" (OuterVolumeSpecName: "home") pod "06c179e2-fb3b-4618-a137-5e8218d38578" (UID: "06c179e2-fb3b-4618-a137-5e8218d38578"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:07.411003 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.410916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-model-cache" (OuterVolumeSpecName: "model-cache") pod "06c179e2-fb3b-4618-a137-5e8218d38578" (UID: "06c179e2-fb3b-4618-a137-5e8218d38578"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:07.411172 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.411144    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06c179e2-fb3b-4618-a137-5e8218d38578" (UID: "06c179e2-fb3b-4618-a137-5e8218d38578"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:07.411172 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.411157    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:07.411361 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.411192    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:07.413180 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.413155    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-dshm" (OuterVolumeSpecName: "dshm") pod "06c179e2-fb3b-4618-a137-5e8218d38578" (UID: "06c179e2-fb3b-4618-a137-5e8218d38578"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:11:07.413282 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.413261    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c179e2-fb3b-4618-a137-5e8218d38578-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06c179e2-fb3b-4618-a137-5e8218d38578" (UID: "06c179e2-fb3b-4618-a137-5e8218d38578"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:11:07.413282 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.413265    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c179e2-fb3b-4618-a137-5e8218d38578-kube-api-access-qzhpc" (OuterVolumeSpecName: "kube-api-access-qzhpc") pod "06c179e2-fb3b-4618-a137-5e8218d38578" (UID: "06c179e2-fb3b-4618-a137-5e8218d38578"). InnerVolumeSpecName "kube-api-access-qzhpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:11:07.512528 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.512508    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:07.512611 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.512548    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c179e2-fb3b-4618-a137-5e8218d38578-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:07.512611 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.512558    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzhpc\" (UniqueName: \"kubernetes.io/projected/06c179e2-fb3b-4618-a137-5e8218d38578-kube-api-access-qzhpc\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:07.512611 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.512565    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c179e2-fb3b-4618-a137-5e8218d38578-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:07.745734 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.745707    2578 generic.go:356] "Generic (PLEG): container finished" podID="06c179e2-fb3b-4618-a137-5e8218d38578" containerID="04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522" exitCode=0
Sep 08 15:11:07.745828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.745753    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"
Sep 08 15:11:07.745828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.745759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p" event={"ID":"06c179e2-fb3b-4618-a137-5e8218d38578","Type":"ContainerDied","Data":"04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522"}
Sep 08 15:11:07.745828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.745788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p" event={"ID":"06c179e2-fb3b-4618-a137-5e8218d38578","Type":"ContainerDied","Data":"dad0903d6f4463010dfcd963891e257a82f4e1ede03355eec8706ff27d4c2f67"}
Sep 08 15:11:07.745828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.745803    2578 scope.go:117] "RemoveContainer" containerID="04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522"
Sep 08 15:11:07.754689 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.754674    2578 scope.go:117] "RemoveContainer" containerID="04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522"
Sep 08 15:11:07.754977 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:11:07.754961    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522\": container with ID starting with 04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522 not found: ID does not exist" containerID="04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522"
Sep 08 15:11:07.755035 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.754989    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522"} err="failed to get container status \"04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522\": rpc error: code = NotFound desc = could not find container \"04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522\": container with ID starting with 04bedc72228d6bd0a701523c39dc19e576371a0415429450f45830aee98d6522 not found: ID does not exist"
Sep 08 15:11:07.782068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.782018    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"]
Sep 08 15:11:07.786269 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.786250    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-6b2b2e/autores-model-6b2b2e-kserve-5d97d674bc-znl7p"]
Sep 08 15:11:07.831501 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.831480    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c179e2-fb3b-4618-a137-5e8218d38578" path="/var/lib/kubelet/pods/06c179e2-fb3b-4618-a137-5e8218d38578/volumes"
Sep 08 15:11:07.831850 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:07.831837    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77545d2-c445-45cb-8aa6-b5265ffd612c" path="/var/lib/kubelet/pods/c77545d2-c445-45cb-8aa6-b5265ffd612c/volumes"
Sep 08 15:11:11.708023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.707982    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:11.738042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.738011    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"]
Sep 08 15:11:11.738259 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.738233    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2" podUID="60793971-d1f1-4c97-a30d-1a5bf6a3f162" containerName="payload-processing" containerID="cri-o://e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e" gracePeriod=30
Sep 08 15:11:11.742277 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.742251    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"]
Sep 08 15:11:11.742598 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.742568    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk" podUID="d90f72c2-c416-47c0-98f6-bc0582de7405" containerName="payload-pre-processing" containerID="cri-o://498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb" gracePeriod=30
Sep 08 15:11:11.778021 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.777983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:11.889162 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:11.889124    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:12.011979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.011957    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:11:12.014775 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.014759    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:11:12.025293 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.025269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:12.096200 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.096175    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:12.146478 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.146457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg4ck\" (UniqueName: \"kubernetes.io/projected/d90f72c2-c416-47c0-98f6-bc0582de7405-kube-api-access-sg4ck\") pod \"d90f72c2-c416-47c0-98f6-bc0582de7405\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") "
Sep 08 15:11:12.146603 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.146492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90f72c2-c416-47c0-98f6-bc0582de7405-plugins-config-volume\") pod \"d90f72c2-c416-47c0-98f6-bc0582de7405\" (UID: \"d90f72c2-c416-47c0-98f6-bc0582de7405\") "
Sep 08 15:11:12.146603 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.146565    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nblp8\" (UniqueName: \"kubernetes.io/projected/60793971-d1f1-4c97-a30d-1a5bf6a3f162-kube-api-access-nblp8\") pod \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") "
Sep 08 15:11:12.146682 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.146614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60793971-d1f1-4c97-a30d-1a5bf6a3f162-plugins-config-volume\") pod \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\" (UID: \"60793971-d1f1-4c97-a30d-1a5bf6a3f162\") "
Sep 08 15:11:12.146957 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.146935    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90f72c2-c416-47c0-98f6-bc0582de7405-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d90f72c2-c416-47c0-98f6-bc0582de7405" (UID: "d90f72c2-c416-47c0-98f6-bc0582de7405"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:11:12.147056 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.146993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60793971-d1f1-4c97-a30d-1a5bf6a3f162-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60793971-d1f1-4c97-a30d-1a5bf6a3f162" (UID: "60793971-d1f1-4c97-a30d-1a5bf6a3f162"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:11:12.148702 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.148679    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90f72c2-c416-47c0-98f6-bc0582de7405-kube-api-access-sg4ck" (OuterVolumeSpecName: "kube-api-access-sg4ck") pod "d90f72c2-c416-47c0-98f6-bc0582de7405" (UID: "d90f72c2-c416-47c0-98f6-bc0582de7405"). InnerVolumeSpecName "kube-api-access-sg4ck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:11:12.148797 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.148773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60793971-d1f1-4c97-a30d-1a5bf6a3f162-kube-api-access-nblp8" (OuterVolumeSpecName: "kube-api-access-nblp8") pod "60793971-d1f1-4c97-a30d-1a5bf6a3f162" (UID: "60793971-d1f1-4c97-a30d-1a5bf6a3f162"). InnerVolumeSpecName "kube-api-access-nblp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:11:12.248152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.248099    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60793971-d1f1-4c97-a30d-1a5bf6a3f162-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:12.248152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.248117    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg4ck\" (UniqueName: \"kubernetes.io/projected/d90f72c2-c416-47c0-98f6-bc0582de7405-kube-api-access-sg4ck\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:12.248152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.248126    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d90f72c2-c416-47c0-98f6-bc0582de7405-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:12.248152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.248137    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nblp8\" (UniqueName: \"kubernetes.io/projected/60793971-d1f1-4c97-a30d-1a5bf6a3f162-kube-api-access-nblp8\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:11:12.753537 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.753495    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:12.776720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.776687    2578 generic.go:356] "Generic (PLEG): container finished" podID="60793971-d1f1-4c97-a30d-1a5bf6a3f162" containerID="e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e" exitCode=0
Sep 08 15:11:12.776821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.776738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"
Sep 08 15:11:12.776821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.776787    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2" event={"ID":"60793971-d1f1-4c97-a30d-1a5bf6a3f162","Type":"ContainerDied","Data":"e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e"}
Sep 08 15:11:12.776939 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.776850    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2" event={"ID":"60793971-d1f1-4c97-a30d-1a5bf6a3f162","Type":"ContainerDied","Data":"3e70f17e77ca507de56bcf1b89cd8ec5553237ece3a5350c247cd0bd1ce6fc05"}
Sep 08 15:11:12.776939 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.776875    2578 scope.go:117] "RemoveContainer" containerID="e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e"
Sep 08 15:11:12.778311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.778167    2578 generic.go:356] "Generic (PLEG): container finished" podID="d90f72c2-c416-47c0-98f6-bc0582de7405" containerID="498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb" exitCode=0
Sep 08 15:11:12.778311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.778192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk" event={"ID":"d90f72c2-c416-47c0-98f6-bc0582de7405","Type":"ContainerDied","Data":"498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb"}
Sep 08 15:11:12.778311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.778215    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"
Sep 08 15:11:12.778311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.778225    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk" event={"ID":"d90f72c2-c416-47c0-98f6-bc0582de7405","Type":"ContainerDied","Data":"9be26e2fed730b9baf14a1937fa2eb661ded4a1fadf1d4eec69f6fb45998cb29"}
Sep 08 15:11:12.788092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.788072    2578 scope.go:117] "RemoveContainer" containerID="e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e"
Sep 08 15:11:12.788650 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:11:12.788379    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e\": container with ID starting with e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e not found: ID does not exist" containerID="e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e"
Sep 08 15:11:12.788650 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.788403    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e"} err="failed to get container status \"e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e\": rpc error: code = NotFound desc = could not find container \"e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e\": container with ID starting with e8a0e429827646961d65292c8caa964fd5b55693bbd69ffc394973e17117a72e not found: ID does not exist"
Sep 08 15:11:12.788650 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.788420    2578 scope.go:117] "RemoveContainer" containerID="498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb"
Sep 08 15:11:12.797069 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.797051    2578 scope.go:117] "RemoveContainer" containerID="498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb"
Sep 08 15:11:12.797351 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:11:12.797333    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb\": container with ID starting with 498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb not found: ID does not exist" containerID="498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb"
Sep 08 15:11:12.797401 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.797360    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb"} err="failed to get container status \"498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb\": rpc error: code = NotFound desc = could not find container \"498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb\": container with ID starting with 498f9dc94c72e4e585f2eee6f45f1bb944c902b089d14b43415cf452931364bb not found: ID does not exist"
Sep 08 15:11:12.804224 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.804203    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"]
Sep 08 15:11:12.809694 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.809673    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-9b84af36-84bf79d9d9-h9nj2"]
Sep 08 15:11:12.820662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.820641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"]
Sep 08 15:11:12.826514 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.826496    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9b84af36-6567fc6f7d-94vvk"]
Sep 08 15:11:12.828399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.828383    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:12.901190 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:12.901171    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:13.831043 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:13.831008    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60793971-d1f1-4c97-a30d-1a5bf6a3f162" path="/var/lib/kubelet/pods/60793971-d1f1-4c97-a30d-1a5bf6a3f162/volumes"
Sep 08 15:11:13.831378 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:13.831324    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d90f72c2-c416-47c0-98f6-bc0582de7405" path="/var/lib/kubelet/pods/d90f72c2-c416-47c0-98f6-bc0582de7405/volumes"
Sep 08 15:11:14.370814 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:14.370784    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:20.266985 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:20.266952    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:20.367969 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:20.367942    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:20.466877 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:20.466855    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:20.567511 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:20.567448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:20.981157 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:20.981088    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:31.702324 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702281    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"]
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702651    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c179e2-fb3b-4618-a137-5e8218d38578" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702665    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c179e2-fb3b-4618-a137-5e8218d38578" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702672    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d90f72c2-c416-47c0-98f6-bc0582de7405" containerName="payload-pre-processing"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702678    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d90f72c2-c416-47c0-98f6-bc0582de7405" containerName="payload-pre-processing"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702689    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60793971-d1f1-4c97-a30d-1a5bf6a3f162" containerName="payload-processing"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702694    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="60793971-d1f1-4c97-a30d-1a5bf6a3f162" containerName="payload-processing"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702707    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99777269-2a1b-49ae-b1bc-548af29974fd" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702712    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="99777269-2a1b-49ae-b1bc-548af29974fd" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702721    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77545d2-c445-45cb-8aa6-b5265ffd612c" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702726    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77545d2-c445-45cb-8aa6-b5265ffd612c" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702733    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57e857c-954d-4901-b298-a3279df49f70" containerName="maas-api"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702739    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57e857c-954d-4901-b298-a3279df49f70" containerName="maas-api"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702791    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77545d2-c445-45cb-8aa6-b5265ffd612c" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702799    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57e857c-954d-4901-b298-a3279df49f70" containerName="maas-api"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702806    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="60793971-d1f1-4c97-a30d-1a5bf6a3f162" containerName="payload-processing"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702812    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="99777269-2a1b-49ae-b1bc-548af29974fd" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702818    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c179e2-fb3b-4618-a137-5e8218d38578" containerName="main"
Sep 08 15:11:31.702828 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.702824    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d90f72c2-c416-47c0-98f6-bc0582de7405" containerName="payload-pre-processing"
Sep 08 15:11:31.706058 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.706036    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.709729 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.709704    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eba68d\"/\"default-dockercfg-mbdvq\""
Sep 08 15:11:31.709853 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.709704    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eba68d\"/\"kube-root-ca.crt\""
Sep 08 15:11:31.709853 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.709706    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-eba68d\"/\"test-model-eba68d-kserve-self-signed-certs\""
Sep 08 15:11:31.709853 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.709753    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-eba68d\"/\"openshift-service-ca.crt\""
Sep 08 15:11:31.715932 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.715906    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"]
Sep 08 15:11:31.784464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.784440    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:31.891688 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.891665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-dshm\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.891788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.891696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-home\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.891788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.891717    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78s9\" (UniqueName: \"kubernetes.io/projected/d436716d-4b1a-43e6-a9cd-8e3301ff886d-kube-api-access-f78s9\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.891788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.891739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tls-certs\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.891900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.891798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tmp-dir\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.891900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.891829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-model-cache\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992456    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-dshm\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-home\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992507    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78s9\" (UniqueName: \"kubernetes.io/projected/d436716d-4b1a-43e6-a9cd-8e3301ff886d-kube-api-access-f78s9\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tls-certs\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992693    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tmp-dir\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992799 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-model-cache\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.992946 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.992925    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-home\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.993085 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.993062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tmp-dir\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.993154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.993071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-model-cache\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.994962 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.994945    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-dshm\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:31.995311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:31.995294    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tls-certs\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:32.001843 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:32.001817    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78s9\" (UniqueName: \"kubernetes.io/projected/d436716d-4b1a-43e6-a9cd-8e3301ff886d-kube-api-access-f78s9\") pod \"test-model-eba68d-kserve-64674887d9-zvl8g\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") " pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:32.016680 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:32.016662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:32.296776 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:32.296753    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"]
Sep 08 15:11:32.298724 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:11:32.298698    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd436716d_4b1a_43e6_a9cd_8e3301ff886d.slice/crio-935edb1a78b0d946262b2dcf25fc93bb3c4b40864165f072bdbe7dc5bd2b0c2b WatchSource:0}: Error finding container 935edb1a78b0d946262b2dcf25fc93bb3c4b40864165f072bdbe7dc5bd2b0c2b: Status 404 returned error can't find the container with id 935edb1a78b0d946262b2dcf25fc93bb3c4b40864165f072bdbe7dc5bd2b0c2b
Sep 08 15:11:32.872774 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:32.872740    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g" event={"ID":"d436716d-4b1a-43e6-a9cd-8e3301ff886d","Type":"ContainerStarted","Data":"7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72"}
Sep 08 15:11:32.872774 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:32.872777    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g" event={"ID":"d436716d-4b1a-43e6-a9cd-8e3301ff886d","Type":"ContainerStarted","Data":"935edb1a78b0d946262b2dcf25fc93bb3c4b40864165f072bdbe7dc5bd2b0c2b"}
Sep 08 15:11:32.894675 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:32.894623    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g" podStartSLOduration=1.894604688 podStartE2EDuration="1.894604688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:11:32.894213689 +0000 UTC m=+971.545637997" watchObservedRunningTime="2026-09-08 15:11:32.894604688 +0000 UTC m=+971.546028999"
Sep 08 15:11:33.537481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:33.537445    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:33.786390 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:33.786335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:34.242435 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:34.242404    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:34.560640 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:34.560610    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:34.709258 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:34.709227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:37.882908 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:37.882873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:37.974967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:37.974926    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:39.056723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.056646    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:39.158481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.158448    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:39.544477 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.544446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:39.652586 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.652519    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:39.694552 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.694494    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"]
Sep 08 15:11:39.697092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.697072    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.702239 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.702220    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ac43b2\""
Sep 08 15:11:39.727428 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.727406    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"]
Sep 08 15:11:39.753196 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.753171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41118743-1df9-460f-84ea-c9d623f6329f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.753323 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.753213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qz7c\" (UniqueName: \"kubernetes.io/projected/41118743-1df9-460f-84ea-c9d623f6329f-kube-api-access-4qz7c\") pod \"payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.755228 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.755207    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"]
Sep 08 15:11:39.757544 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.757512    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:39.760162 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.760147    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ac43b2-dockercfg-x7hsl\""
Sep 08 15:11:39.772841 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.772808    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"]
Sep 08 15:11:39.853974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.853910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41118743-1df9-460f-84ea-c9d623f6329f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.853974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.853947    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdjj\" (UniqueName: \"kubernetes.io/projected/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-kube-api-access-jvdjj\") pod \"payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:39.853974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.853975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qz7c\" (UniqueName: \"kubernetes.io/projected/41118743-1df9-460f-84ea-c9d623f6329f-kube-api-access-4qz7c\") pod \"payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.854244 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.854031    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:39.854668 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.854647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41118743-1df9-460f-84ea-c9d623f6329f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.864673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.864653    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qz7c\" (UniqueName: \"kubernetes.io/projected/41118743-1df9-460f-84ea-c9d623f6329f-kube-api-access-4qz7c\") pod \"payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") " pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:39.954580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.954557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdjj\" (UniqueName: \"kubernetes.io/projected/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-kube-api-access-jvdjj\") pod \"payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:39.954678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.954621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:39.955196 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.955180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:39.963625 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:39.963599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdjj\" (UniqueName: \"kubernetes.io/projected/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-kube-api-access-jvdjj\") pod \"payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:40.007366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.007344    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:40.066432 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.066403    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:40.280888 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.280866    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"]
Sep 08 15:11:40.282883 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:11:40.282856    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41118743_1df9_460f_84ea_c9d623f6329f.slice/crio-986659ff01126c69ecfe955b39473b3811fb529761795a565090b8a566422b5d WatchSource:0}: Error finding container 986659ff01126c69ecfe955b39473b3811fb529761795a565090b8a566422b5d: Status 404 returned error can't find the container with id 986659ff01126c69ecfe955b39473b3811fb529761795a565090b8a566422b5d
Sep 08 15:11:40.339883 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.339860    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"]
Sep 08 15:11:40.342001 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:11:40.341978    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3f4f63_ec85_4a44_96b4_a530e2ab6c4d.slice/crio-8e7ed827978bc2a774180d1ec09a05fa707cd00eed99bf37c077b77f560407c3 WatchSource:0}: Error finding container 8e7ed827978bc2a774180d1ec09a05fa707cd00eed99bf37c077b77f560407c3: Status 404 returned error can't find the container with id 8e7ed827978bc2a774180d1ec09a05fa707cd00eed99bf37c077b77f560407c3
Sep 08 15:11:40.906905 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.906880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647" event={"ID":"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d","Type":"ContainerStarted","Data":"fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0"}
Sep 08 15:11:40.907049 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.906923    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647" event={"ID":"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d","Type":"ContainerStarted","Data":"8e7ed827978bc2a774180d1ec09a05fa707cd00eed99bf37c077b77f560407c3"}
Sep 08 15:11:40.907049 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.907008    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:40.908664 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.908640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt" event={"ID":"41118743-1df9-460f-84ea-c9d623f6329f","Type":"ContainerStarted","Data":"709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3"}
Sep 08 15:11:40.908785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.908687    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt" event={"ID":"41118743-1df9-460f-84ea-c9d623f6329f","Type":"ContainerStarted","Data":"986659ff01126c69ecfe955b39473b3811fb529761795a565090b8a566422b5d"}
Sep 08 15:11:40.908785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.908718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:40.926760 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.926721    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647" podStartSLOduration=1.926707714 podStartE2EDuration="1.926707714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:11:40.924690315 +0000 UTC m=+979.576114623" watchObservedRunningTime="2026-09-08 15:11:40.926707714 +0000 UTC m=+979.578132022"
Sep 08 15:11:40.942834 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:40.942796    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt" podStartSLOduration=1.94278398 podStartE2EDuration="1.94278398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:11:40.941987367 +0000 UTC m=+979.593411676" watchObservedRunningTime="2026-09-08 15:11:40.94278398 +0000 UTC m=+979.594208359"
Sep 08 15:11:42.017714 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:42.017682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:42.017714 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:42.017720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:42.030721 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:42.030695    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:42.929038 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:42.929004    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:11:46.986882 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:46.986845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:51.916441 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:51.916406    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:11:51.917088 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:51.916557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:11:52.987299 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:52.987265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:53.047859 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:53.047827    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:53.145382 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:53.145354    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:11:53.272202 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:11:53.272176    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:02.871181 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:02.871144    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:09.114976 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:09.114943    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:14.789905 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:14.789862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:14.889271 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:14.889242    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:14.987806 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:14.987776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:17.998012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:17.997982    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"]
Sep 08 15:12:18.002492 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.002473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.006094 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.006072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ac43b2\"/\"default-dockercfg-tjsts\""
Sep 08 15:12:18.006388 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.006371    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ac43b2\"/\"kube-root-ca.crt\""
Sep 08 15:12:18.006436 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.006376    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-ac43b2\"/\"explicit-model-ac43b2-kserve-self-signed-certs\""
Sep 08 15:12:18.006436 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.006375    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-ac43b2\"/\"openshift-service-ca.crt\""
Sep 08 15:12:18.015407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.015384    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"]
Sep 08 15:12:18.024931 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.024906    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:18.160201 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.160175    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-home\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.160309 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.160208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tls-certs\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.160309 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.160229    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-model-cache\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.160309 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.160261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-dshm\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.160411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.160303    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vffw\" (UniqueName: \"kubernetes.io/projected/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-kube-api-access-4vffw\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.160411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.160388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tmp-dir\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261057 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261036    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-home\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tls-certs\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-model-cache\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-dshm\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261149    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vffw\" (UniqueName: \"kubernetes.io/projected/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-kube-api-access-4vffw\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261368 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261238    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tmp-dir\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261479 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-home\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261692 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-model-cache\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.261807 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.261703    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tmp-dir\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.263580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.263560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-dshm\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.263864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.263847    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tls-certs\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.269708 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.269689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vffw\" (UniqueName: \"kubernetes.io/projected/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-kube-api-access-4vffw\") pod \"explicit-model-ac43b2-kserve-c8b549fff-qt9mt\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") " pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.312659 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.312638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:18.591059 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.590990    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"]
Sep 08 15:12:18.593818 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:12:18.593791    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf54937cb_35e6_42d1_aac3_db9aa5eb9b83.slice/crio-d76c93779e51bf9ae57cc3a37cc4698c34d8ac717d9219fbf6a5e8d4e3ce140b WatchSource:0}: Error finding container d76c93779e51bf9ae57cc3a37cc4698c34d8ac717d9219fbf6a5e8d4e3ce140b: Status 404 returned error can't find the container with id d76c93779e51bf9ae57cc3a37cc4698c34d8ac717d9219fbf6a5e8d4e3ce140b
Sep 08 15:12:18.895429 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:18.895361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:19.095983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:19.095946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt" event={"ID":"f54937cb-35e6-42d1-aac3-db9aa5eb9b83","Type":"ContainerStarted","Data":"67cd04feda6fe0bbbd643b4765269d5eef218f41992e6b4de815b7a7bb92fbb9"}
Sep 08 15:12:19.095983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:19.095995    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt" event={"ID":"f54937cb-35e6-42d1-aac3-db9aa5eb9b83","Type":"ContainerStarted","Data":"d76c93779e51bf9ae57cc3a37cc4698c34d8ac717d9219fbf6a5e8d4e3ce140b"}
Sep 08 15:12:19.114444 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:19.114398    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt" podStartSLOduration=2.114382796 podStartE2EDuration="2.114382796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:19.113406481 +0000 UTC m=+1017.764830791" watchObservedRunningTime="2026-09-08 15:12:19.114382796 +0000 UTC m=+1017.765807166"
Sep 08 15:12:21.924431 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:21.924403    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"]
Sep 08 15:12:21.947860 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:21.947830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:25.506968 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:25.506933    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:26.309924 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:26.309890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:28.313696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:28.313653    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:32.033241 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.033206    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:32.136448 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.136422    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:32.241462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.241433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:32.337866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.337802    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:32.668099 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.668002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:32.703592 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.703566    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"]
Sep 08 15:12:32.703806 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.703779    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt" podUID="41118743-1df9-460f-84ea-c9d623f6329f" containerName="payload-processing" containerID="cri-o://709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3" gracePeriod=30
Sep 08 15:12:32.707969 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.707941    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"]
Sep 08 15:12:32.708207 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.708179    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647" podUID="aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" containerName="payload-pre-processing" containerID="cri-o://fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0" gracePeriod=30
Sep 08 15:12:32.953598 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.953575    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:12:32.974006 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:32.973988    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:12:33.080965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.080942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvdjj\" (UniqueName: \"kubernetes.io/projected/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-kube-api-access-jvdjj\") pod \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") "
Sep 08 15:12:33.081397 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.080981    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41118743-1df9-460f-84ea-c9d623f6329f-plugins-config-volume\") pod \"41118743-1df9-460f-84ea-c9d623f6329f\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") "
Sep 08 15:12:33.081397 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.080998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qz7c\" (UniqueName: \"kubernetes.io/projected/41118743-1df9-460f-84ea-c9d623f6329f-kube-api-access-4qz7c\") pod \"41118743-1df9-460f-84ea-c9d623f6329f\" (UID: \"41118743-1df9-460f-84ea-c9d623f6329f\") "
Sep 08 15:12:33.081397 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.081043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-plugins-config-volume\") pod \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\" (UID: \"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d\") "
Sep 08 15:12:33.081397 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.081297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41118743-1df9-460f-84ea-c9d623f6329f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "41118743-1df9-460f-84ea-c9d623f6329f" (UID: "41118743-1df9-460f-84ea-c9d623f6329f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:12:33.081591 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.081445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" (UID: "aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:12:33.083216 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.083192    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41118743-1df9-460f-84ea-c9d623f6329f-kube-api-access-4qz7c" (OuterVolumeSpecName: "kube-api-access-4qz7c") pod "41118743-1df9-460f-84ea-c9d623f6329f" (UID: "41118743-1df9-460f-84ea-c9d623f6329f"). InnerVolumeSpecName "kube-api-access-4qz7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:12:33.083216 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.083197    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-kube-api-access-jvdjj" (OuterVolumeSpecName: "kube-api-access-jvdjj") pod "aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" (UID: "aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d"). InnerVolumeSpecName "kube-api-access-jvdjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:12:33.163410 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.163386    2578 generic.go:356] "Generic (PLEG): container finished" podID="aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" containerID="fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0" exitCode=0
Sep 08 15:12:33.163554 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.163432    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"
Sep 08 15:12:33.163554 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.163461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647" event={"ID":"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d","Type":"ContainerDied","Data":"fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0"}
Sep 08 15:12:33.163554 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.163504    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647" event={"ID":"aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d","Type":"ContainerDied","Data":"8e7ed827978bc2a774180d1ec09a05fa707cd00eed99bf37c077b77f560407c3"}
Sep 08 15:12:33.163554 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.163546    2578 scope.go:117] "RemoveContainer" containerID="fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0"
Sep 08 15:12:33.164910 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.164891    2578 generic.go:356] "Generic (PLEG): container finished" podID="41118743-1df9-460f-84ea-c9d623f6329f" containerID="709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3" exitCode=0
Sep 08 15:12:33.165006 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.164941    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt" event={"ID":"41118743-1df9-460f-84ea-c9d623f6329f","Type":"ContainerDied","Data":"709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3"}
Sep 08 15:12:33.165068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.165003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt" event={"ID":"41118743-1df9-460f-84ea-c9d623f6329f","Type":"ContainerDied","Data":"986659ff01126c69ecfe955b39473b3811fb529761795a565090b8a566422b5d"}
Sep 08 15:12:33.165133 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.165087    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"
Sep 08 15:12:33.173182 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.173161    2578 scope.go:117] "RemoveContainer" containerID="fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0"
Sep 08 15:12:33.173468 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:12:33.173448    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0\": container with ID starting with fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0 not found: ID does not exist" containerID="fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0"
Sep 08 15:12:33.173569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.173474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0"} err="failed to get container status \"fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0\": rpc error: code = NotFound desc = could not find container \"fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0\": container with ID starting with fc98e64ca7fdf42ec3a87a45daa5cc5838d26f056adf60a1d9d77d84c56da8e0 not found: ID does not exist"
Sep 08 15:12:33.173569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.173492    2578 scope.go:117] "RemoveContainer" containerID="709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3"
Sep 08 15:12:33.182024 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.182000    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:33.182124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.182028    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvdjj\" (UniqueName: \"kubernetes.io/projected/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d-kube-api-access-jvdjj\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:33.182124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.182042    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41118743-1df9-460f-84ea-c9d623f6329f-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:33.182124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.182054    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qz7c\" (UniqueName: \"kubernetes.io/projected/41118743-1df9-460f-84ea-c9d623f6329f-kube-api-access-4qz7c\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:33.184329 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.184313    2578 scope.go:117] "RemoveContainer" containerID="709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3"
Sep 08 15:12:33.184608 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:12:33.184589    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3\": container with ID starting with 709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3 not found: ID does not exist" containerID="709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3"
Sep 08 15:12:33.184697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.184612    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3"} err="failed to get container status \"709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3\": rpc error: code = NotFound desc = could not find container \"709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3\": container with ID starting with 709468cc7e7282e5106d6a9e9e53344d7561f16bf76da4c05f1fa5d9b8dcece3 not found: ID does not exist"
Sep 08 15:12:33.186952 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.186933    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"]
Sep 08 15:12:33.198411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.198388    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ac43b2-5dc7b98cd-dp647"]
Sep 08 15:12:33.209399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.209380    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"]
Sep 08 15:12:33.221957 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.221938    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ac43b2-5549486cfc-htdwt"]
Sep 08 15:12:33.831145 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.831119    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41118743-1df9-460f-84ea-c9d623f6329f" path="/var/lib/kubelet/pods/41118743-1df9-460f-84ea-c9d623f6329f/volumes"
Sep 08 15:12:33.831456 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:33.831444    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" path="/var/lib/kubelet/pods/aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d/volumes"
Sep 08 15:12:35.834750 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:35.834699    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:35.937455 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:35.937407    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:36.944002 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:36.943969    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt" podUID="f54937cb-35e6-42d1-aac3-db9aa5eb9b83" containerName="main" containerID="cri-o://67cd04feda6fe0bbbd643b4765269d5eef218f41992e6b4de815b7a7bb92fbb9" gracePeriod=45
Sep 08 15:12:37.186266 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.186238    2578 generic.go:356] "Generic (PLEG): container finished" podID="f54937cb-35e6-42d1-aac3-db9aa5eb9b83" containerID="67cd04feda6fe0bbbd643b4765269d5eef218f41992e6b4de815b7a7bb92fbb9" exitCode=0
Sep 08 15:12:37.186393 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.186314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt" event={"ID":"f54937cb-35e6-42d1-aac3-db9aa5eb9b83","Type":"ContainerDied","Data":"67cd04feda6fe0bbbd643b4765269d5eef218f41992e6b4de815b7a7bb92fbb9"}
Sep 08 15:12:37.186393 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.186357    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt" event={"ID":"f54937cb-35e6-42d1-aac3-db9aa5eb9b83","Type":"ContainerDied","Data":"d76c93779e51bf9ae57cc3a37cc4698c34d8ac717d9219fbf6a5e8d4e3ce140b"}
Sep 08 15:12:37.186393 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.186370    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d76c93779e51bf9ae57cc3a37cc4698c34d8ac717d9219fbf6a5e8d4e3ce140b"
Sep 08 15:12:37.193263 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.193246    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:37.212572 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212494    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tls-certs\") pod \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") "
Sep 08 15:12:37.212678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212574    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-home\") pod \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") "
Sep 08 15:12:37.212678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212592    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-model-cache\") pod \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") "
Sep 08 15:12:37.212678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vffw\" (UniqueName: \"kubernetes.io/projected/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-kube-api-access-4vffw\") pod \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") "
Sep 08 15:12:37.212678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212652    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tmp-dir\") pod \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") "
Sep 08 15:12:37.212678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-dshm\") pod \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\" (UID: \"f54937cb-35e6-42d1-aac3-db9aa5eb9b83\") "
Sep 08 15:12:37.212968 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212833    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-home" (OuterVolumeSpecName: "home") pod "f54937cb-35e6-42d1-aac3-db9aa5eb9b83" (UID: "f54937cb-35e6-42d1-aac3-db9aa5eb9b83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:12:37.212968 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212844    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-model-cache" (OuterVolumeSpecName: "model-cache") pod "f54937cb-35e6-42d1-aac3-db9aa5eb9b83" (UID: "f54937cb-35e6-42d1-aac3-db9aa5eb9b83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:12:37.213083 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212970    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:37.213083 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.212986    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:37.213261 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.213212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f54937cb-35e6-42d1-aac3-db9aa5eb9b83" (UID: "f54937cb-35e6-42d1-aac3-db9aa5eb9b83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:12:37.215592 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.215558    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-dshm" (OuterVolumeSpecName: "dshm") pod "f54937cb-35e6-42d1-aac3-db9aa5eb9b83" (UID: "f54937cb-35e6-42d1-aac3-db9aa5eb9b83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:12:37.215719 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.215623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f54937cb-35e6-42d1-aac3-db9aa5eb9b83" (UID: "f54937cb-35e6-42d1-aac3-db9aa5eb9b83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:12:37.215719 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.215635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-kube-api-access-4vffw" (OuterVolumeSpecName: "kube-api-access-4vffw") pod "f54937cb-35e6-42d1-aac3-db9aa5eb9b83" (UID: "f54937cb-35e6-42d1-aac3-db9aa5eb9b83"). InnerVolumeSpecName "kube-api-access-4vffw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:12:37.313395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.313373    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vffw\" (UniqueName: \"kubernetes.io/projected/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-kube-api-access-4vffw\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:37.313395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.313394    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:37.313554 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.313405    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:37.313554 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:37.313412    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f54937cb-35e6-42d1-aac3-db9aa5eb9b83-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:12:38.191410 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:38.191343    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"
Sep 08 15:12:38.208112 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:38.208084    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"]
Sep 08 15:12:38.211843 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:38.211818    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-ac43b2/explicit-model-ac43b2-kserve-c8b549fff-qt9mt"]
Sep 08 15:12:39.252758 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:39.252727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:39.352983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:39.352955    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:39.507673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:39.507647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:39.832165 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:39.832093    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f54937cb-35e6-42d1-aac3-db9aa5eb9b83" path="/var/lib/kubelet/pods/f54937cb-35e6-42d1-aac3-db9aa5eb9b83/volumes"
Sep 08 15:12:40.048805 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:40.048773    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:40.314328 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:40.314292    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:40.399359 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:40.399326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:41.572938 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:41.572901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:41.685909 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:41.685874    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:41.785713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:41.785672    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:41.884555 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:41.884455    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:42.309901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:42.309869    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:48.475122 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.475085    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:48.497690 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.497665    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"]
Sep 08 15:12:48.497998 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.497986    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41118743-1df9-460f-84ea-c9d623f6329f" containerName="payload-processing"
Sep 08 15:12:48.498042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498002    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41118743-1df9-460f-84ea-c9d623f6329f" containerName="payload-processing"
Sep 08 15:12:48.498042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498017    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f54937cb-35e6-42d1-aac3-db9aa5eb9b83" containerName="main"
Sep 08 15:12:48.498042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498023    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f54937cb-35e6-42d1-aac3-db9aa5eb9b83" containerName="main"
Sep 08 15:12:48.498042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498036    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" containerName="payload-pre-processing"
Sep 08 15:12:48.498042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498042    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" containerName="payload-pre-processing"
Sep 08 15:12:48.498194 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498100    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa3f4f63-ec85-4a44-96b4-a530e2ab6c4d" containerName="payload-pre-processing"
Sep 08 15:12:48.498194 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498112    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41118743-1df9-460f-84ea-c9d623f6329f" containerName="payload-processing"
Sep 08 15:12:48.498194 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.498118    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f54937cb-35e6-42d1-aac3-db9aa5eb9b83" containerName="main"
Sep 08 15:12:48.500899 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.500883    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.503369 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.503349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e3e2a7\"/\"noten-model-e3e2a7-kserve-self-signed-certs\""
Sep 08 15:12:48.504295 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.504274    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e3e2a7\"/\"openshift-service-ca.crt\""
Sep 08 15:12:48.504394 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.504299    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e3e2a7\"/\"kube-root-ca.crt\""
Sep 08 15:12:48.504394 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.504349    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e3e2a7\"/\"default-dockercfg-mzzb5\""
Sep 08 15:12:48.517713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.517694    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"]
Sep 08 15:12:48.573665 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.573634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:48.601407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.601380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.601513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.601414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-tmp-dir\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.601513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.601457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-home\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.601513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.601495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-model-cache\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.601681 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.601609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-dshm\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.601681 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.601636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wcps\" (UniqueName: \"kubernetes.io/projected/5ede0563-806e-49f7-990b-de3027d60499-kube-api-access-6wcps\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.685459 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.685432    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:48.702940 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.702917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-dshm\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703038 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.702944    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wcps\" (UniqueName: \"kubernetes.io/projected/5ede0563-806e-49f7-990b-de3027d60499-kube-api-access-6wcps\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703038 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.702973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703038 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.702990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-tmp-dir\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703038 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.703025    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-home\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.703060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-model-cache\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703387 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.703359    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-tmp-dir\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703492 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.703406    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-model-cache\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.703492 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.703450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-home\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.705427 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.705407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-dshm\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.705662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.705644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.711628 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.711608    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wcps\" (UniqueName: \"kubernetes.io/projected/5ede0563-806e-49f7-990b-de3027d60499-kube-api-access-6wcps\") pod \"noten-model-e3e2a7-kserve-697798675f-f2t25\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") " pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:48.809723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:48.809698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:49.043110 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:49.043078    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:49.100234 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:49.100214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"]
Sep 08 15:12:49.101900 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:12:49.101871    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ede0563_806e_49f7_990b_de3027d60499.slice/crio-721b939488fa579b435e874a448c3cae1533711313804e61af5dc4c959b15a83 WatchSource:0}: Error finding container 721b939488fa579b435e874a448c3cae1533711313804e61af5dc4c959b15a83: Status 404 returned error can't find the container with id 721b939488fa579b435e874a448c3cae1533711313804e61af5dc4c959b15a83
Sep 08 15:12:49.246592 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:49.246553    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25" event={"ID":"5ede0563-806e-49f7-990b-de3027d60499","Type":"ContainerStarted","Data":"c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2"}
Sep 08 15:12:49.246753 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:49.246604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25" event={"ID":"5ede0563-806e-49f7-990b-de3027d60499","Type":"ContainerStarted","Data":"721b939488fa579b435e874a448c3cae1533711313804e61af5dc4c959b15a83"}
Sep 08 15:12:49.276751 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:49.276703    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25" podStartSLOduration=1.276687397 podStartE2EDuration="1.276687397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:49.274948564 +0000 UTC m=+1047.926372875" watchObservedRunningTime="2026-09-08 15:12:49.276687397 +0000 UTC m=+1047.928111707"
Sep 08 15:12:49.488899 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:49.488835    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:51.498927 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:51.498898    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:52.389085 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:52.389046    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:52.479716 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:52.479678    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:53.552720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:53.552691    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:53.660337 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:53.660307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:54.076502 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.076465    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:54.172269 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.172216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:54.209263 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.209235    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"]
Sep 08 15:12:54.212723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.212706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.218875 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.218847    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b04cf783\""
Sep 08 15:12:54.218985 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.218944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b04cf783-dockercfg-qjrvm\""
Sep 08 15:12:54.228670 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.228651    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"]
Sep 08 15:12:54.250866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.250844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgl22\" (UniqueName: \"kubernetes.io/projected/b56c4a8a-5946-4f78-a203-df99cf8a5c13-kube-api-access-wgl22\") pod \"payload-processing-e2e-mt-b04cf783-769c684657-w29xh\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") " pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.250965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.250875    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c4a8a-5946-4f78-a203-df99cf8a5c13-plugins-config-volume\") pod \"payload-processing-e2e-mt-b04cf783-769c684657-w29xh\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") " pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.266352 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.266329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:12:54.266931 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.266913    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"]
Sep 08 15:12:54.270211 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.270193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.282456 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.282437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"]
Sep 08 15:12:54.351695 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.351607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgl22\" (UniqueName: \"kubernetes.io/projected/b56c4a8a-5946-4f78-a203-df99cf8a5c13-kube-api-access-wgl22\") pod \"payload-processing-e2e-mt-b04cf783-769c684657-w29xh\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") " pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.351695 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.351643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c4a8a-5946-4f78-a203-df99cf8a5c13-plugins-config-volume\") pod \"payload-processing-e2e-mt-b04cf783-769c684657-w29xh\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") " pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.351947 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.351848    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wr6m\" (UniqueName: \"kubernetes.io/projected/c15552a6-1fb9-41fa-9fbf-41138ab5bead-kube-api-access-4wr6m\") pod \"payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.351947 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.351892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15552a6-1fb9-41fa-9fbf-41138ab5bead-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.352580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.352562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c4a8a-5946-4f78-a203-df99cf8a5c13-plugins-config-volume\") pod \"payload-processing-e2e-mt-b04cf783-769c684657-w29xh\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") " pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.361281 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.361255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgl22\" (UniqueName: \"kubernetes.io/projected/b56c4a8a-5946-4f78-a203-df99cf8a5c13-kube-api-access-wgl22\") pod \"payload-processing-e2e-mt-b04cf783-769c684657-w29xh\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") " pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.452464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.452428    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wr6m\" (UniqueName: \"kubernetes.io/projected/c15552a6-1fb9-41fa-9fbf-41138ab5bead-kube-api-access-4wr6m\") pod \"payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.452464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.452466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15552a6-1fb9-41fa-9fbf-41138ab5bead-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.453202 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.453182    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15552a6-1fb9-41fa-9fbf-41138ab5bead-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.460886 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.460855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wr6m\" (UniqueName: \"kubernetes.io/projected/c15552a6-1fb9-41fa-9fbf-41138ab5bead-kube-api-access-4wr6m\") pod \"payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.523712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.523652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:54.580062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.579797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:54.800377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.800355    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"]
Sep 08 15:12:54.802242 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:12:54.802217    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56c4a8a_5946_4f78_a203_df99cf8a5c13.slice/crio-21863436137d67ccc1af8debc2c774dd3644e83f937e880b6f9b26c57d8c97cb WatchSource:0}: Error finding container 21863436137d67ccc1af8debc2c774dd3644e83f937e880b6f9b26c57d8c97cb: Status 404 returned error can't find the container with id 21863436137d67ccc1af8debc2c774dd3644e83f937e880b6f9b26c57d8c97cb
Sep 08 15:12:54.860669 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:54.860649    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"]
Sep 08 15:12:54.861945 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:12:54.861923    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15552a6_1fb9_41fa_9fbf_41138ab5bead.slice/crio-b155d528bac3dfc01c8584191003c7226baa34b531c6d3914d2407db53601c63 WatchSource:0}: Error finding container b155d528bac3dfc01c8584191003c7226baa34b531c6d3914d2407db53601c63: Status 404 returned error can't find the container with id b155d528bac3dfc01c8584191003c7226baa34b531c6d3914d2407db53601c63
Sep 08 15:12:55.277070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.277036    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh" event={"ID":"b56c4a8a-5946-4f78-a203-df99cf8a5c13","Type":"ContainerStarted","Data":"16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde"}
Sep 08 15:12:55.277244 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.277095    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh" event={"ID":"b56c4a8a-5946-4f78-a203-df99cf8a5c13","Type":"ContainerStarted","Data":"21863436137d67ccc1af8debc2c774dd3644e83f937e880b6f9b26c57d8c97cb"}
Sep 08 15:12:55.277244 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.277124    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:12:55.278793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.278769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z" event={"ID":"c15552a6-1fb9-41fa-9fbf-41138ab5bead","Type":"ContainerStarted","Data":"e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2"}
Sep 08 15:12:55.278901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.278817    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z" event={"ID":"c15552a6-1fb9-41fa-9fbf-41138ab5bead","Type":"ContainerStarted","Data":"b155d528bac3dfc01c8584191003c7226baa34b531c6d3914d2407db53601c63"}
Sep 08 15:12:55.278901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.278847    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:12:55.293696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.293645    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh" podStartSLOduration=1.293627099 podStartE2EDuration="1.293627099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:55.293207868 +0000 UTC m=+1053.944632178" watchObservedRunningTime="2026-09-08 15:12:55.293627099 +0000 UTC m=+1053.945051409"
Sep 08 15:12:55.311717 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:55.311668    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z" podStartSLOduration=1.311651644 podStartE2EDuration="1.311651644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:12:55.311061301 +0000 UTC m=+1053.962485613" watchObservedRunningTime="2026-09-08 15:12:55.311651644 +0000 UTC m=+1053.963075962"
Sep 08 15:12:58.810084 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:58.810053    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:58.810456 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:58.810113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:58.822739 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:58.822708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:59.308222 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:59.308186    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:12:59.899336 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:59.899310    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"]
Sep 08 15:12:59.926475 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:12:59.926446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:00.121872 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:00.121828    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:00.973212 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:00.973182    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"]
Sep 08 15:13:00.996420 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:00.996386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:01.116687 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:01.116656    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:01.344568 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:01.344535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:01.409323 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:01.409294    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-self-signed-certs: secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:01.409490 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:01.409362    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs podName:5ede0563-806e-49f7-990b-de3027d60499 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:01.909347255 +0000 UTC m=+1060.560771542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs") pod "noten-model-e3e2a7-kserve-697798675f-f2t25" (UID: "5ede0563-806e-49f7-990b-de3027d60499") : secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:01.482298 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:01.482269    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:01.914094 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:01.914066    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-self-signed-certs: secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:01.914244 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:01.914132    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs podName:5ede0563-806e-49f7-990b-de3027d60499 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:02.914114282 +0000 UTC m=+1061.565538580 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs") pod "noten-model-e3e2a7-kserve-697798675f-f2t25" (UID: "5ede0563-806e-49f7-990b-de3027d60499") : secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:02.922161 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:02.922124    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-self-signed-certs: secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:02.922618 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:02.922193    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs podName:5ede0563-806e-49f7-990b-de3027d60499 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:04.922176646 +0000 UTC m=+1063.573600934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs") pod "noten-model-e3e2a7-kserve-697798675f-f2t25" (UID: "5ede0563-806e-49f7-990b-de3027d60499") : secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:04.939209 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:04.939174    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-self-signed-certs: secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:04.939690 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:04.939249    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs podName:5ede0563-806e-49f7-990b-de3027d60499 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:08.939234414 +0000 UTC m=+1067.590658700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs") pod "noten-model-e3e2a7-kserve-697798675f-f2t25" (UID: "5ede0563-806e-49f7-990b-de3027d60499") : secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:06.285571 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:06.285495    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:13:06.286011 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:06.285629    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:13:06.546059 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:06.545974    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:08.974077 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:08.974036    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-self-signed-certs: secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:08.974497 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:08.974121    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs podName:5ede0563-806e-49f7-990b-de3027d60499 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:16.974106418 +0000 UTC m=+1075.625530705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs") pod "noten-model-e3e2a7-kserve-697798675f-f2t25" (UID: "5ede0563-806e-49f7-990b-de3027d60499") : secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:10.472193 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.472159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"]
Sep 08 15:13:10.472621 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.472448    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" podUID="7ca25c10-005a-4f12-bfc5-f61534d3f321" containerName="maas-api" containerID="cri-o://b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525" gracePeriod=30
Sep 08 15:13:10.504109 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.504086    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:10.924983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.924963    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:13:10.990813 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.990793    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca25c10-005a-4f12-bfc5-f61534d3f321-openshift-service-ca\") pod \"7ca25c10-005a-4f12-bfc5-f61534d3f321\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") "
Sep 08 15:13:10.990924 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.990856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rchpx\" (UniqueName: \"kubernetes.io/projected/7ca25c10-005a-4f12-bfc5-f61534d3f321-kube-api-access-rchpx\") pod \"7ca25c10-005a-4f12-bfc5-f61534d3f321\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") "
Sep 08 15:13:10.990924 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.990910    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-metrics-certs\") pod \"7ca25c10-005a-4f12-bfc5-f61534d3f321\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") "
Sep 08 15:13:10.991012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.990961    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls\") pod \"7ca25c10-005a-4f12-bfc5-f61534d3f321\" (UID: \"7ca25c10-005a-4f12-bfc5-f61534d3f321\") "
Sep 08 15:13:10.991187 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.991164    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca25c10-005a-4f12-bfc5-f61534d3f321-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ca25c10-005a-4f12-bfc5-f61534d3f321" (UID: "7ca25c10-005a-4f12-bfc5-f61534d3f321"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:13:10.991303 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.991290    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca25c10-005a-4f12-bfc5-f61534d3f321-openshift-service-ca\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:10.993027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.993004    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca25c10-005a-4f12-bfc5-f61534d3f321-kube-api-access-rchpx" (OuterVolumeSpecName: "kube-api-access-rchpx") pod "7ca25c10-005a-4f12-bfc5-f61534d3f321" (UID: "7ca25c10-005a-4f12-bfc5-f61534d3f321"). InnerVolumeSpecName "kube-api-access-rchpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:13:10.993116 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.993091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ca25c10-005a-4f12-bfc5-f61534d3f321" (UID: "7ca25c10-005a-4f12-bfc5-f61534d3f321"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:13:10.993116 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:10.993105    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ca25c10-005a-4f12-bfc5-f61534d3f321" (UID: "7ca25c10-005a-4f12-bfc5-f61534d3f321"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:13:11.092294 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.092265    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rchpx\" (UniqueName: \"kubernetes.io/projected/7ca25c10-005a-4f12-bfc5-f61534d3f321-kube-api-access-rchpx\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:11.092294 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.092289    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:11.092416 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.092302    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca25c10-005a-4f12-bfc5-f61534d3f321-maas-api-tls\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:11.362968 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.362938    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ca25c10-005a-4f12-bfc5-f61534d3f321" containerID="b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525" exitCode=0
Sep 08 15:13:11.363089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.363006    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"
Sep 08 15:13:11.363089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.363027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" event={"ID":"7ca25c10-005a-4f12-bfc5-f61534d3f321","Type":"ContainerDied","Data":"b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525"}
Sep 08 15:13:11.363089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.363082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w" event={"ID":"7ca25c10-005a-4f12-bfc5-f61534d3f321","Type":"ContainerDied","Data":"9d80661c66412afbafb64888a32a1885f6cf8e0f038ee7e95a9a35b8a5d0e418"}
Sep 08 15:13:11.363216 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.363105    2578 scope.go:117] "RemoveContainer" containerID="b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525"
Sep 08 15:13:11.372365 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.372350    2578 scope.go:117] "RemoveContainer" containerID="b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525"
Sep 08 15:13:11.372656 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:11.372638    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525\": container with ID starting with b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525 not found: ID does not exist" containerID="b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525"
Sep 08 15:13:11.372717 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.372664    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525"} err="failed to get container status \"b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525\": rpc error: code = NotFound desc = could not find container \"b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525\": container with ID starting with b9a7c6d60dd4d6e4c896d04ad5d66aa8138206f191ae677921df189eae1e3525 not found: ID does not exist"
Sep 08 15:13:11.385703 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.385681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"]
Sep 08 15:13:11.389015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.388994    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a1592f-54cdf775b6-kn59w"]
Sep 08 15:13:11.832262 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:11.832232    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca25c10-005a-4f12-bfc5-f61534d3f321" path="/var/lib/kubelet/pods/7ca25c10-005a-4f12-bfc5-f61534d3f321/volumes"
Sep 08 15:13:13.097438 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:13.097406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:13.199320 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:13.199284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:14.551157 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:14.551128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:14.919097 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:14.919008    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g" podUID="d436716d-4b1a-43e6-a9cd-8e3301ff886d" containerName="main" containerID="cri-o://7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72" gracePeriod=45
Sep 08 15:13:15.163942 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.163921    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:13:15.327917 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.327889    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f78s9\" (UniqueName: \"kubernetes.io/projected/d436716d-4b1a-43e6-a9cd-8e3301ff886d-kube-api-access-f78s9\") pod \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") "
Sep 08 15:13:15.328070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.327924    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-dshm\") pod \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") "
Sep 08 15:13:15.328070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.327947    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tls-certs\") pod \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") "
Sep 08 15:13:15.328070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.327978    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tmp-dir\") pod \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") "
Sep 08 15:13:15.328070 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-home\") pod \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") "
Sep 08 15:13:15.328307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-model-cache\") pod \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\" (UID: \"d436716d-4b1a-43e6-a9cd-8e3301ff886d\") "
Sep 08 15:13:15.328307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328264    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d436716d-4b1a-43e6-a9cd-8e3301ff886d" (UID: "d436716d-4b1a-43e6-a9cd-8e3301ff886d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:15.328425 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328335    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-home" (OuterVolumeSpecName: "home") pod "d436716d-4b1a-43e6-a9cd-8e3301ff886d" (UID: "d436716d-4b1a-43e6-a9cd-8e3301ff886d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:15.328481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328425    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:15.328481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328443    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:15.328481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.328464    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d436716d-4b1a-43e6-a9cd-8e3301ff886d" (UID: "d436716d-4b1a-43e6-a9cd-8e3301ff886d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:15.330372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.330317    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d436716d-4b1a-43e6-a9cd-8e3301ff886d" (UID: "d436716d-4b1a-43e6-a9cd-8e3301ff886d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:13:15.330372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.330357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d436716d-4b1a-43e6-a9cd-8e3301ff886d-kube-api-access-f78s9" (OuterVolumeSpecName: "kube-api-access-f78s9") pod "d436716d-4b1a-43e6-a9cd-8e3301ff886d" (UID: "d436716d-4b1a-43e6-a9cd-8e3301ff886d"). InnerVolumeSpecName "kube-api-access-f78s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:13:15.330505 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.330416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-dshm" (OuterVolumeSpecName: "dshm") pod "d436716d-4b1a-43e6-a9cd-8e3301ff886d" (UID: "d436716d-4b1a-43e6-a9cd-8e3301ff886d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:15.386637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.386609    2578 generic.go:356] "Generic (PLEG): container finished" podID="d436716d-4b1a-43e6-a9cd-8e3301ff886d" containerID="7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72" exitCode=0
Sep 08 15:13:15.386740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.386656    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"
Sep 08 15:13:15.386740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.386671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g" event={"ID":"d436716d-4b1a-43e6-a9cd-8e3301ff886d","Type":"ContainerDied","Data":"7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72"}
Sep 08 15:13:15.386740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.386705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g" event={"ID":"d436716d-4b1a-43e6-a9cd-8e3301ff886d","Type":"ContainerDied","Data":"935edb1a78b0d946262b2dcf25fc93bb3c4b40864165f072bdbe7dc5bd2b0c2b"}
Sep 08 15:13:15.386740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.386722    2578 scope.go:117] "RemoveContainer" containerID="7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72"
Sep 08 15:13:15.395550 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.395536    2578 scope.go:117] "RemoveContainer" containerID="7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72"
Sep 08 15:13:15.395814 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:15.395794    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72\": container with ID starting with 7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72 not found: ID does not exist" containerID="7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72"
Sep 08 15:13:15.395866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.395822    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72"} err="failed to get container status \"7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72\": rpc error: code = NotFound desc = could not find container \"7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72\": container with ID starting with 7a1a5eabaf1ff4400b2bede37028c817dc38f064fbf784457a574af39b6e0f72 not found: ID does not exist"
Sep 08 15:13:15.406791 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.406767    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"]
Sep 08 15:13:15.410335 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.410316    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-eba68d/test-model-eba68d-kserve-64674887d9-zvl8g"]
Sep 08 15:13:15.429723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.429699    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:15.429723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.429718    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d436716d-4b1a-43e6-a9cd-8e3301ff886d-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:15.429723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.429726    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d436716d-4b1a-43e6-a9cd-8e3301ff886d-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:15.429897 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.429735    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f78s9\" (UniqueName: \"kubernetes.io/projected/d436716d-4b1a-43e6-a9cd-8e3301ff886d-kube-api-access-f78s9\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:15.831427 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:15.831382    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d436716d-4b1a-43e6-a9cd-8e3301ff886d" path="/var/lib/kubelet/pods/d436716d-4b1a-43e6-a9cd-8e3301ff886d/volumes"
Sep 08 15:13:17.042076 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:17.042045    2578 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-self-signed-certs: secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:17.042745 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:17.042343    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs podName:5ede0563-806e-49f7-990b-de3027d60499 nodeName:}" failed. No retries permitted until 2026-09-08 15:13:33.042307442 +0000 UTC m=+1091.693731741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs") pod "noten-model-e3e2a7-kserve-697798675f-f2t25" (UID: "5ede0563-806e-49f7-990b-de3027d60499") : secret "noten-model-e3e2a7-kserve-self-signed-certs" not found
Sep 08 15:13:17.076311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.076286    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:17.150825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.150803    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:17.337374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.337301    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25" podUID="5ede0563-806e-49f7-990b-de3027d60499" containerName="main" containerID="cri-o://c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2" gracePeriod=45
Sep 08 15:13:17.415917 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.415885    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:17.587939 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.587885    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:13:17.749059 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749032    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-model-cache\") pod \"5ede0563-806e-49f7-990b-de3027d60499\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") "
Sep 08 15:13:17.749236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wcps\" (UniqueName: \"kubernetes.io/projected/5ede0563-806e-49f7-990b-de3027d60499-kube-api-access-6wcps\") pod \"5ede0563-806e-49f7-990b-de3027d60499\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") "
Sep 08 15:13:17.749236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749124    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-tmp-dir\") pod \"5ede0563-806e-49f7-990b-de3027d60499\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") "
Sep 08 15:13:17.749236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749154    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs\") pod \"5ede0563-806e-49f7-990b-de3027d60499\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") "
Sep 08 15:13:17.749236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-dshm\") pod \"5ede0563-806e-49f7-990b-de3027d60499\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") "
Sep 08 15:13:17.749236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749204    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-home\") pod \"5ede0563-806e-49f7-990b-de3027d60499\" (UID: \"5ede0563-806e-49f7-990b-de3027d60499\") "
Sep 08 15:13:17.749502 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-model-cache" (OuterVolumeSpecName: "model-cache") pod "5ede0563-806e-49f7-990b-de3027d60499" (UID: "5ede0563-806e-49f7-990b-de3027d60499"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:17.749502 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5ede0563-806e-49f7-990b-de3027d60499" (UID: "5ede0563-806e-49f7-990b-de3027d60499"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:17.749502 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749466    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:17.749673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.749572    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-home" (OuterVolumeSpecName: "home") pod "5ede0563-806e-49f7-990b-de3027d60499" (UID: "5ede0563-806e-49f7-990b-de3027d60499"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:17.751598 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.751573    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-dshm" (OuterVolumeSpecName: "dshm") pod "5ede0563-806e-49f7-990b-de3027d60499" (UID: "5ede0563-806e-49f7-990b-de3027d60499"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:13:17.751710 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.751688    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5ede0563-806e-49f7-990b-de3027d60499" (UID: "5ede0563-806e-49f7-990b-de3027d60499"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:13:17.751761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.751690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ede0563-806e-49f7-990b-de3027d60499-kube-api-access-6wcps" (OuterVolumeSpecName: "kube-api-access-6wcps") pod "5ede0563-806e-49f7-990b-de3027d60499" (UID: "5ede0563-806e-49f7-990b-de3027d60499"). InnerVolumeSpecName "kube-api-access-6wcps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:13:17.849867 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.849841    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wcps\" (UniqueName: \"kubernetes.io/projected/5ede0563-806e-49f7-990b-de3027d60499-kube-api-access-6wcps\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:17.849867 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.849866    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:17.850028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.849878    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ede0563-806e-49f7-990b-de3027d60499-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:17.850028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.849885    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:17.850028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:17.849893    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ede0563-806e-49f7-990b-de3027d60499-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:18.146967 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.146890    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:18.263232 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.263204    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:18.403580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.403477    2578 generic.go:356] "Generic (PLEG): container finished" podID="5ede0563-806e-49f7-990b-de3027d60499" containerID="c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2" exitCode=0
Sep 08 15:13:18.403580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.403518    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"
Sep 08 15:13:18.403580 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.403571    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25" event={"ID":"5ede0563-806e-49f7-990b-de3027d60499","Type":"ContainerDied","Data":"c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2"}
Sep 08 15:13:18.403809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.403616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25" event={"ID":"5ede0563-806e-49f7-990b-de3027d60499","Type":"ContainerDied","Data":"721b939488fa579b435e874a448c3cae1533711313804e61af5dc4c959b15a83"}
Sep 08 15:13:18.403809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.403635    2578 scope.go:117] "RemoveContainer" containerID="c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2"
Sep 08 15:13:18.409355 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.409334    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:18.412273 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.412254    2578 scope.go:117] "RemoveContainer" containerID="c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2"
Sep 08 15:13:18.412553 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:18.412510    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2\": container with ID starting with c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2 not found: ID does not exist" containerID="c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2"
Sep 08 15:13:18.412597 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.412566    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2"} err="failed to get container status \"c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2\": rpc error: code = NotFound desc = could not find container \"c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2\": container with ID starting with c390d9a654e2e61eb6a59c72208165f873dcde0760ec81576a9c5154c89620f2 not found: ID does not exist"
Sep 08 15:13:18.426703 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.426673    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"]
Sep 08 15:13:18.428841 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.428812    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e3e2a7/noten-model-e3e2a7-kserve-697798675f-f2t25"]
Sep 08 15:13:18.601447 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.601414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:18.947656 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:18.947630    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:19.832226 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:19.832191    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ede0563-806e-49f7-990b-de3027d60499" path="/var/lib/kubelet/pods/5ede0563-806e-49f7-990b-de3027d60499/volumes"
Sep 08 15:13:24.720464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:24.720433    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:24.821091 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:24.821061    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:25.183198 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:25.183165    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:29.068864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:29.068829    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:29.832242 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:29.832210    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:29.914902 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:29.914866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:30.016866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:30.016797    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:30.097266 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:30.097199    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:30.169034 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:30.169002    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:30.536676 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:30.536647    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:35.194395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.194361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:35.783112 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.783073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:35.811628 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.811604    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"]
Sep 08 15:13:35.811988 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.811955    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh" podUID="b56c4a8a-5946-4f78-a203-df99cf8a5c13" containerName="payload-processing" containerID="cri-o://16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde" gracePeriod=30
Sep 08 15:13:35.820399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.820376    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"]
Sep 08 15:13:35.820635 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.820611    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z" podUID="c15552a6-1fb9-41fa-9fbf-41138ab5bead" containerName="payload-pre-processing" containerID="cri-o://e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2" gracePeriod=30
Sep 08 15:13:35.854632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.854567    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:35.927253 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.927227    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:35.998826 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:35.998796    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:36.085543 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.085503    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:13:36.090992 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.090974    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgl22\" (UniqueName: \"kubernetes.io/projected/b56c4a8a-5946-4f78-a203-df99cf8a5c13-kube-api-access-wgl22\") pod \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") "
Sep 08 15:13:36.091071 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.091001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c4a8a-5946-4f78-a203-df99cf8a5c13-plugins-config-volume\") pod \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\" (UID: \"b56c4a8a-5946-4f78-a203-df99cf8a5c13\") "
Sep 08 15:13:36.091373 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.091357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56c4a8a-5946-4f78-a203-df99cf8a5c13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b56c4a8a-5946-4f78-a203-df99cf8a5c13" (UID: "b56c4a8a-5946-4f78-a203-df99cf8a5c13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:13:36.091895 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.091879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:13:36.092982 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.092964    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56c4a8a-5946-4f78-a203-df99cf8a5c13-kube-api-access-wgl22" (OuterVolumeSpecName: "kube-api-access-wgl22") pod "b56c4a8a-5946-4f78-a203-df99cf8a5c13" (UID: "b56c4a8a-5946-4f78-a203-df99cf8a5c13"). InnerVolumeSpecName "kube-api-access-wgl22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:13:36.192306 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.192286    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15552a6-1fb9-41fa-9fbf-41138ab5bead-plugins-config-volume\") pod \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") "
Sep 08 15:13:36.192415 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.192340    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4wr6m\" (UniqueName: \"kubernetes.io/projected/c15552a6-1fb9-41fa-9fbf-41138ab5bead-kube-api-access-4wr6m\") pod \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\" (UID: \"c15552a6-1fb9-41fa-9fbf-41138ab5bead\") "
Sep 08 15:13:36.192494 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.192483    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgl22\" (UniqueName: \"kubernetes.io/projected/b56c4a8a-5946-4f78-a203-df99cf8a5c13-kube-api-access-wgl22\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:36.192556 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.192496    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b56c4a8a-5946-4f78-a203-df99cf8a5c13-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:36.192641 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.192622    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c15552a6-1fb9-41fa-9fbf-41138ab5bead-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c15552a6-1fb9-41fa-9fbf-41138ab5bead" (UID: "c15552a6-1fb9-41fa-9fbf-41138ab5bead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:13:36.194364 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.194345    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c15552a6-1fb9-41fa-9fbf-41138ab5bead-kube-api-access-4wr6m" (OuterVolumeSpecName: "kube-api-access-4wr6m") pod "c15552a6-1fb9-41fa-9fbf-41138ab5bead" (UID: "c15552a6-1fb9-41fa-9fbf-41138ab5bead"). InnerVolumeSpecName "kube-api-access-4wr6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:13:36.293103 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.293081    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wr6m\" (UniqueName: \"kubernetes.io/projected/c15552a6-1fb9-41fa-9fbf-41138ab5bead-kube-api-access-4wr6m\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:36.293103 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.293101    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c15552a6-1fb9-41fa-9fbf-41138ab5bead-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:13:36.430385 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.430317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:36.488260 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.488220    2578 generic.go:356] "Generic (PLEG): container finished" podID="b56c4a8a-5946-4f78-a203-df99cf8a5c13" containerID="16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde" exitCode=0
Sep 08 15:13:36.488366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.488255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh" event={"ID":"b56c4a8a-5946-4f78-a203-df99cf8a5c13","Type":"ContainerDied","Data":"16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde"}
Sep 08 15:13:36.488366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.488271    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"
Sep 08 15:13:36.488366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.488311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh" event={"ID":"b56c4a8a-5946-4f78-a203-df99cf8a5c13","Type":"ContainerDied","Data":"21863436137d67ccc1af8debc2c774dd3644e83f937e880b6f9b26c57d8c97cb"}
Sep 08 15:13:36.488366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.488342    2578 scope.go:117] "RemoveContainer" containerID="16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde"
Sep 08 15:13:36.489923 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.489901    2578 generic.go:356] "Generic (PLEG): container finished" podID="c15552a6-1fb9-41fa-9fbf-41138ab5bead" containerID="e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2" exitCode=0
Sep 08 15:13:36.490018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.489934    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"
Sep 08 15:13:36.490018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.489982    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z" event={"ID":"c15552a6-1fb9-41fa-9fbf-41138ab5bead","Type":"ContainerDied","Data":"e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2"}
Sep 08 15:13:36.490097 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.490023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z" event={"ID":"c15552a6-1fb9-41fa-9fbf-41138ab5bead","Type":"ContainerDied","Data":"b155d528bac3dfc01c8584191003c7226baa34b531c6d3914d2407db53601c63"}
Sep 08 15:13:36.499477 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.499267    2578 scope.go:117] "RemoveContainer" containerID="16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde"
Sep 08 15:13:36.499893 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:36.499867    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde\": container with ID starting with 16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde not found: ID does not exist" containerID="16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde"
Sep 08 15:13:36.499999 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.499902    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde"} err="failed to get container status \"16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde\": rpc error: code = NotFound desc = could not find container \"16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde\": container with ID starting with 16e31a3863f037cdc8f8f954e598ba76fa009ddb373696cb66daae1d31e86cde not found: ID does not exist"
Sep 08 15:13:36.499999 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.499928    2578 scope.go:117] "RemoveContainer" containerID="e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2"
Sep 08 15:13:36.508138 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.508125    2578 scope.go:117] "RemoveContainer" containerID="e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2"
Sep 08 15:13:36.508389 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:13:36.508373    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2\": container with ID starting with e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2 not found: ID does not exist" containerID="e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2"
Sep 08 15:13:36.508442 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.508394    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2"} err="failed to get container status \"e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2\": rpc error: code = NotFound desc = could not find container \"e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2\": container with ID starting with e796d78bd50c5bacd508dce306de80914f8909ecc5f01f94a6b1ff231217e3b2 not found: ID does not exist"
Sep 08 15:13:36.514243 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.514223    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"]
Sep 08 15:13:36.518139 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.518121    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b04cf783-769c684657-w29xh"]
Sep 08 15:13:36.528560 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.528543    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"]
Sep 08 15:13:36.532384 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:36.532364    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-b04cf783-68d4694898-zvk7z"]
Sep 08 15:13:37.380061 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:37.380018    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:37.445599 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:37.445574    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:37.674229 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:37.674157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:37.832298 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:37.832263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56c4a8a-5946-4f78-a203-df99cf8a5c13" path="/var/lib/kubelet/pods/b56c4a8a-5946-4f78-a203-df99cf8a5c13/volumes"
Sep 08 15:13:37.832637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:37.832621    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c15552a6-1fb9-41fa-9fbf-41138ab5bead" path="/var/lib/kubelet/pods/c15552a6-1fb9-41fa-9fbf-41138ab5bead/volumes"
Sep 08 15:13:38.317228 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:38.317195    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:38.784310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:38.784278    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:39.470414 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:39.470386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:41.761300 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:41.761268    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:41.851292 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:41.851262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:42.573866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:42.573830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:42.673337 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:42.673304    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:42.834604 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:42.834510    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:43.180381 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.180307    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:43.295219 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.295157    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:43.462829 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.462766    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:43.826289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.826262    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:43.926449 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.926416    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:43.957359 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957323    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"]
Sep 08 15:13:43.957679 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957664    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ede0563-806e-49f7-990b-de3027d60499" containerName="main"
Sep 08 15:13:43.957679 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ede0563-806e-49f7-990b-de3027d60499" containerName="main"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957700    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca25c10-005a-4f12-bfc5-f61534d3f321" containerName="maas-api"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957706    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca25c10-005a-4f12-bfc5-f61534d3f321" containerName="maas-api"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957712    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56c4a8a-5946-4f78-a203-df99cf8a5c13" containerName="payload-processing"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957722    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56c4a8a-5946-4f78-a203-df99cf8a5c13" containerName="payload-processing"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957736    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c15552a6-1fb9-41fa-9fbf-41138ab5bead" containerName="payload-pre-processing"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957742    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c15552a6-1fb9-41fa-9fbf-41138ab5bead" containerName="payload-pre-processing"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957750    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d436716d-4b1a-43e6-a9cd-8e3301ff886d" containerName="main"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957754    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d436716d-4b1a-43e6-a9cd-8e3301ff886d" containerName="main"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957809    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56c4a8a-5946-4f78-a203-df99cf8a5c13" containerName="payload-processing"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957819    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca25c10-005a-4f12-bfc5-f61534d3f321" containerName="maas-api"
Sep 08 15:13:43.957818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957826    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d436716d-4b1a-43e6-a9cd-8e3301ff886d" containerName="main"
Sep 08 15:13:43.958343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957830    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c15552a6-1fb9-41fa-9fbf-41138ab5bead" containerName="payload-pre-processing"
Sep 08 15:13:43.958343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.957837    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ede0563-806e-49f7-990b-de3027d60499" containerName="main"
Sep 08 15:13:43.962454 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.962425    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:43.969581 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.965094    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-ffa941\""
Sep 08 15:13:43.969581 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.965412    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-ffa941-dockercfg-6gv54\""
Sep 08 15:13:43.982191 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:43.982156    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"]
Sep 08 15:13:44.047625 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.047593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqp7k\" (UniqueName: \"kubernetes.io/projected/156d3494-a178-4015-a22b-1b98ca825d43-kube-api-access-tqp7k\") pod \"payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.047765 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.047671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3494-a178-4015-a22b-1b98ca825d43-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.148973 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.148893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqp7k\" (UniqueName: \"kubernetes.io/projected/156d3494-a178-4015-a22b-1b98ca825d43-kube-api-access-tqp7k\") pod \"payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.148973 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.148947    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3494-a178-4015-a22b-1b98ca825d43-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.149597 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.149578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3494-a178-4015-a22b-1b98ca825d43-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.157377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.157353    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqp7k\" (UniqueName: \"kubernetes.io/projected/156d3494-a178-4015-a22b-1b98ca825d43-kube-api-access-tqp7k\") pod \"payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.275409 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.275380    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:44.553472 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.553452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"]
Sep 08 15:13:44.555913 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:13:44.555880    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156d3494_a178_4015_a22b_1b98ca825d43.slice/crio-1edbf91dcadc8fe585e6498e2aff276f260a6f09dd86231ff82f38370d653445 WatchSource:0}: Error finding container 1edbf91dcadc8fe585e6498e2aff276f260a6f09dd86231ff82f38370d653445: Status 404 returned error can't find the container with id 1edbf91dcadc8fe585e6498e2aff276f260a6f09dd86231ff82f38370d653445
Sep 08 15:13:44.646664 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:44.646634    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:45.535962 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:45.535933    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2" event={"ID":"156d3494-a178-4015-a22b-1b98ca825d43","Type":"ContainerStarted","Data":"2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73"}
Sep 08 15:13:45.536415 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:45.535978    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2" event={"ID":"156d3494-a178-4015-a22b-1b98ca825d43","Type":"ContainerStarted","Data":"1edbf91dcadc8fe585e6498e2aff276f260a6f09dd86231ff82f38370d653445"}
Sep 08 15:13:45.536415 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:45.536074    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:45.553900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:45.553846    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2" podStartSLOduration=2.55382952 podStartE2EDuration="2.55382952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:13:45.551824898 +0000 UTC m=+1104.203249208" watchObservedRunningTime="2026-09-08 15:13:45.55382952 +0000 UTC m=+1104.205253833"
Sep 08 15:13:51.519562 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:51.519505    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:51.584173 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:51.584142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:55.520051 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:55.520019    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:55.622322 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:55.622293    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:56.524518 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:56.524486    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:56.542188 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:56.542169    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:13:56.836333 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:56.836265    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:57.156238 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:57.156159    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:59.847585 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:59.847556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:13:59.959620 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:13:59.959592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:01.205797 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:01.205764    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:01.282689 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:01.282661    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:01.453688 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:01.453660    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:01.772060 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:01.772028    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:01.878582 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:01.878548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:05.235478 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:05.235441    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:06.874474 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:06.874443    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:07.026516 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:07.026483    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:07.364965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:07.364929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:07.619547 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:07.619456    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:10.835922 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:10.835892    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:11.098596 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:11.098508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:11.977888 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:11.977858    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:12.079284 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.079251    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:12.464211 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.464181    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:12.537374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.537347    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:12.570011 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.569984    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"]
Sep 08 15:14:12.573514 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.573497    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.576285 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.576257    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7be222\""
Sep 08 15:14:12.584668 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.584648    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"]
Sep 08 15:14:12.615696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.615661    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"]
Sep 08 15:14:12.618961 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.618945    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.621282 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.621262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-7be222-dockercfg-hbd9q\""
Sep 08 15:14:12.632304 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.632283    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"]
Sep 08 15:14:12.668346 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.668326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c86cb89-5596-4106-98cb-f237803a849b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.668468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.668360    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca80a1b-adb6-4d4a-a968-9b597242d223-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.668559 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.668475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfxt\" (UniqueName: \"kubernetes.io/projected/2c86cb89-5596-4106-98cb-f237803a849b-kube-api-access-pvfxt\") pod \"payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.668559 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.668518    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkl8\" (UniqueName: \"kubernetes.io/projected/cca80a1b-adb6-4d4a-a968-9b597242d223-kube-api-access-btkl8\") pod \"payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.769504 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.769483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfxt\" (UniqueName: \"kubernetes.io/projected/2c86cb89-5596-4106-98cb-f237803a849b-kube-api-access-pvfxt\") pod \"payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.769613 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.769517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkl8\" (UniqueName: \"kubernetes.io/projected/cca80a1b-adb6-4d4a-a968-9b597242d223-kube-api-access-btkl8\") pod \"payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.769613 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.769569    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c86cb89-5596-4106-98cb-f237803a849b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.769613 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.769607    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca80a1b-adb6-4d4a-a968-9b597242d223-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.770340 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.770322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca80a1b-adb6-4d4a-a968-9b597242d223-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.770467 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.770446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c86cb89-5596-4106-98cb-f237803a849b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.778919 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.778894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfxt\" (UniqueName: \"kubernetes.io/projected/2c86cb89-5596-4106-98cb-f237803a849b-kube-api-access-pvfxt\") pod \"payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.779235 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.779210    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkl8\" (UniqueName: \"kubernetes.io/projected/cca80a1b-adb6-4d4a-a968-9b597242d223-kube-api-access-btkl8\") pod \"payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:12.884236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.884215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:12.929051 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:12.929025    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:13.157592 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.157569    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"]
Sep 08 15:14:13.160059 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:14:13.160022    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c86cb89_5596_4106_98cb_f237803a849b.slice/crio-e56f6f117a1960ac8a6565c4fce9172b320f96811b72d4b9e822ea62f54c8b23 WatchSource:0}: Error finding container e56f6f117a1960ac8a6565c4fce9172b320f96811b72d4b9e822ea62f54c8b23: Status 404 returned error can't find the container with id e56f6f117a1960ac8a6565c4fce9172b320f96811b72d4b9e822ea62f54c8b23
Sep 08 15:14:13.208092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.208071    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"]
Sep 08 15:14:13.213548 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:14:13.213504    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca80a1b_adb6_4d4a_a968_9b597242d223.slice/crio-938e9f86b71a5759bf0e8f7ff28e92c2559a39ba8bc75a5386ac53bd12103e0a WatchSource:0}: Error finding container 938e9f86b71a5759bf0e8f7ff28e92c2559a39ba8bc75a5386ac53bd12103e0a: Status 404 returned error can't find the container with id 938e9f86b71a5759bf0e8f7ff28e92c2559a39ba8bc75a5386ac53bd12103e0a
Sep 08 15:14:13.663155 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.663111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96" event={"ID":"cca80a1b-adb6-4d4a-a968-9b597242d223","Type":"ContainerStarted","Data":"2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c"}
Sep 08 15:14:13.663360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.663172    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96" event={"ID":"cca80a1b-adb6-4d4a-a968-9b597242d223","Type":"ContainerStarted","Data":"938e9f86b71a5759bf0e8f7ff28e92c2559a39ba8bc75a5386ac53bd12103e0a"}
Sep 08 15:14:13.663360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.663202    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:13.664634 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.664614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8" event={"ID":"2c86cb89-5596-4106-98cb-f237803a849b","Type":"ContainerStarted","Data":"a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff"}
Sep 08 15:14:13.664761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.664649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8" event={"ID":"2c86cb89-5596-4106-98cb-f237803a849b","Type":"ContainerStarted","Data":"e56f6f117a1960ac8a6565c4fce9172b320f96811b72d4b9e822ea62f54c8b23"}
Sep 08 15:14:13.664834 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.664760    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:13.678333 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.678284    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96" podStartSLOduration=1.678267754 podStartE2EDuration="1.678267754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:13.677947357 +0000 UTC m=+1132.329371680" watchObservedRunningTime="2026-09-08 15:14:13.678267754 +0000 UTC m=+1132.329692062"
Sep 08 15:14:13.693857 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:13.693814    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8" podStartSLOduration=1.693804826 podStartE2EDuration="1.693804826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:13.691962308 +0000 UTC m=+1132.343386618" watchObservedRunningTime="2026-09-08 15:14:13.693804826 +0000 UTC m=+1132.345229134"
Sep 08 15:14:22.258395 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:22.258358    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:24.671760 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:24.671728    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:14:24.672256 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:24.671990    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:14:24.792107 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:24.792075    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:25.121704 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:25.121670    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:25.704864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:25.704830    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:25.806732 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:25.806701    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:28.914157 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:28.914120    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:28.991110 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:28.991071    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:30.056856 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.056825    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:30.162206 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.162170    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:30.540861 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.540831    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:30.641842 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.641805    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:30.671016 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.670987    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"]
Sep 08 15:14:30.675711 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.675693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.678074 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.678054    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-65dd51\""
Sep 08 15:14:30.678296 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.678281    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-65dd51-dockercfg-qbfh9\""
Sep 08 15:14:30.686027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.686006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"]
Sep 08 15:14:30.693770 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.693748    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"]
Sep 08 15:14:30.696904 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.696886    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.711602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.711577    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"]
Sep 08 15:14:30.733860 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.733836    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:30.810605 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.810506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9z97\" (UniqueName: \"kubernetes.io/projected/b12412a1-3a49-43f1-8680-8fd68bee245c-kube-api-access-z9z97\") pod \"payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.810810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.810606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8acf928-2d49-4120-aa6b-6147d156b7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.810810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.810649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sjwr\" (UniqueName: \"kubernetes.io/projected/d8acf928-2d49-4120-aa6b-6147d156b7d8-kube-api-access-2sjwr\") pod \"payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.810810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.810720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12412a1-3a49-43f1-8680-8fd68bee245c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.911779 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.911749    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12412a1-3a49-43f1-8680-8fd68bee245c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.911931 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.911806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9z97\" (UniqueName: \"kubernetes.io/projected/b12412a1-3a49-43f1-8680-8fd68bee245c-kube-api-access-z9z97\") pod \"payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.911931 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.911841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8acf928-2d49-4120-aa6b-6147d156b7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.911931 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.911873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sjwr\" (UniqueName: \"kubernetes.io/projected/d8acf928-2d49-4120-aa6b-6147d156b7d8-kube-api-access-2sjwr\") pod \"payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.912428 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.912410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12412a1-3a49-43f1-8680-8fd68bee245c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.912480 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.912451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8acf928-2d49-4120-aa6b-6147d156b7d8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.920922 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.920895    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9z97\" (UniqueName: \"kubernetes.io/projected/b12412a1-3a49-43f1-8680-8fd68bee245c-kube-api-access-z9z97\") pod \"payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:30.921028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.920951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sjwr\" (UniqueName: \"kubernetes.io/projected/d8acf928-2d49-4120-aa6b-6147d156b7d8-kube-api-access-2sjwr\") pod \"payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:30.986788 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:30.986762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:31.018825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.018790    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:31.273177 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.273152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"]
Sep 08 15:14:31.275839 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:14:31.275816    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8acf928_2d49_4120_aa6b_6147d156b7d8.slice/crio-0b039c81af21f67e44f0ea2408c222d6e044a00669ff12a84127c6c9cca28bf6 WatchSource:0}: Error finding container 0b039c81af21f67e44f0ea2408c222d6e044a00669ff12a84127c6c9cca28bf6: Status 404 returned error can't find the container with id 0b039c81af21f67e44f0ea2408c222d6e044a00669ff12a84127c6c9cca28bf6
Sep 08 15:14:31.306272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.306252    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"]
Sep 08 15:14:31.308545 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:14:31.308505    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12412a1_3a49_43f1_8680_8fd68bee245c.slice/crio-b05c9c301b4a953829371eb588f1dba74a637d36217f3643ff60fa5e58c64e0d WatchSource:0}: Error finding container b05c9c301b4a953829371eb588f1dba74a637d36217f3643ff60fa5e58c64e0d: Status 404 returned error can't find the container with id b05c9c301b4a953829371eb588f1dba74a637d36217f3643ff60fa5e58c64e0d
Sep 08 15:14:31.754889 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.754799    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q" event={"ID":"d8acf928-2d49-4120-aa6b-6147d156b7d8","Type":"ContainerStarted","Data":"224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9"}
Sep 08 15:14:31.754889 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.754860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q" event={"ID":"d8acf928-2d49-4120-aa6b-6147d156b7d8","Type":"ContainerStarted","Data":"0b039c81af21f67e44f0ea2408c222d6e044a00669ff12a84127c6c9cca28bf6"}
Sep 08 15:14:31.755110 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.754895    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:31.756306 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.756285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb" event={"ID":"b12412a1-3a49-43f1-8680-8fd68bee245c","Type":"ContainerStarted","Data":"bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9"}
Sep 08 15:14:31.756399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.756324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb" event={"ID":"b12412a1-3a49-43f1-8680-8fd68bee245c","Type":"ContainerStarted","Data":"b05c9c301b4a953829371eb588f1dba74a637d36217f3643ff60fa5e58c64e0d"}
Sep 08 15:14:31.756465 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.756420    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:31.771013 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.770966    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q" podStartSLOduration=1.770948779 podStartE2EDuration="1.770948779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:31.770243874 +0000 UTC m=+1150.421668183" watchObservedRunningTime="2026-09-08 15:14:31.770948779 +0000 UTC m=+1150.422373094"
Sep 08 15:14:31.786451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:31.786403    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb" podStartSLOduration=1.7863866800000001 podStartE2EDuration="1.78638668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:31.7846888 +0000 UTC m=+1150.436113108" watchObservedRunningTime="2026-09-08 15:14:31.78638668 +0000 UTC m=+1150.437810993"
Sep 08 15:14:33.905348 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:33.905317    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:35.784319 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:35.784284    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:38.906767 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.906735    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"]
Sep 08 15:14:38.910416 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.910399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:38.913884 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.913848    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ffa941\"/\"kube-root-ca.crt\""
Sep 08 15:14:38.913884 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.913852    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ffa941\"/\"default-dockercfg-sxj4z\""
Sep 08 15:14:38.914048 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.913857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ffa941\"/\"auth-test-model-ffa941-kserve-self-signed-certs\""
Sep 08 15:14:38.914048 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.913962    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-ffa941\"/\"openshift-service-ca.crt\""
Sep 08 15:14:38.922791 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.922768    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"]
Sep 08 15:14:38.931804 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:38.931781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:39.085951 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.085929    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-dshm\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.086068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.085964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c31545-492b-46fd-83b0-f667706bd5cc-tls-certs\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.086068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.086000    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnj2n\" (UniqueName: \"kubernetes.io/projected/25c31545-492b-46fd-83b0-f667706bd5cc-kube-api-access-rnj2n\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.086068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.086054    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-home\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.086227 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.086082    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-tmp-dir\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.086227 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.086130    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-model-cache\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-dshm\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186437    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c31545-492b-46fd-83b0-f667706bd5cc-tls-certs\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186460    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnj2n\" (UniqueName: \"kubernetes.io/projected/25c31545-492b-46fd-83b0-f667706bd5cc-kube-api-access-rnj2n\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-home\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-tmp-dir\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-model-cache\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186960 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186935    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-home\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.186960 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-model-cache\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.187124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.186998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-tmp-dir\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.188695 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.188671    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-dshm\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.189000 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.188983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c31545-492b-46fd-83b0-f667706bd5cc-tls-certs\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.196899 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.196879    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnj2n\" (UniqueName: \"kubernetes.io/projected/25c31545-492b-46fd-83b0-f667706bd5cc-kube-api-access-rnj2n\") pod \"auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.221195 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.221174    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:39.494003 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.493981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"]
Sep 08 15:14:39.496242 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:14:39.496222    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25c31545_492b_46fd_83b0_f667706bd5cc.slice/crio-15d73a595c3545c1faaa438864623a0203487f9536e212cbd940f4e8a6e0289c WatchSource:0}: Error finding container 15d73a595c3545c1faaa438864623a0203487f9536e212cbd940f4e8a6e0289c: Status 404 returned error can't find the container with id 15d73a595c3545c1faaa438864623a0203487f9536e212cbd940f4e8a6e0289c
Sep 08 15:14:39.802387 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.802353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv" event={"ID":"25c31545-492b-46fd-83b0-f667706bd5cc","Type":"ContainerStarted","Data":"0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e"}
Sep 08 15:14:39.802558 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.802402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv" event={"ID":"25c31545-492b-46fd-83b0-f667706bd5cc","Type":"ContainerStarted","Data":"15d73a595c3545c1faaa438864623a0203487f9536e212cbd940f4e8a6e0289c"}
Sep 08 15:14:39.820080 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:39.820033    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv" podStartSLOduration=1.820015068 podStartE2EDuration="1.820015068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:39.81946058 +0000 UTC m=+1158.470885095" watchObservedRunningTime="2026-09-08 15:14:39.820015068 +0000 UTC m=+1158.471439377"
Sep 08 15:14:42.763696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:42.763662    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:14:42.764240 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:42.763746    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:14:49.221374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:49.221315    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:49.221374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:49.221379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:49.235258 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:49.235233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:49.861590 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:49.861557    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:14:50.963544 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:50.963504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:53.935657 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:53.935615    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:54.697911 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.697876    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:14:54.766813 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.766785    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"]
Sep 08 15:14:54.770273 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.770256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.772768 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.772745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"openshift-service-ca.crt\""
Sep 08 15:14:54.772871 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.772745    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"kube-root-ca.crt\""
Sep 08 15:14:54.773542 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.773505    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"default-dockercfg-zm699\""
Sep 08 15:14:54.773542 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.773519    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"auth-test-model-7be222-kserve-self-signed-certs\""
Sep 08 15:14:54.781225 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.781203    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"]
Sep 08 15:14:54.810026 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.810003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-model-cache\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.810114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.810029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-home\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.810114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.810046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19124252-c993-4fb7-a514-1fa16ba8a3af-tls-certs\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.810114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.810072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64wd4\" (UniqueName: \"kubernetes.io/projected/19124252-c993-4fb7-a514-1fa16ba8a3af-kube-api-access-64wd4\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.810220 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.810148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-dshm\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.810220 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.810194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-tmp-dir\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.910658 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.910634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-dshm\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.910764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.910683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-tmp-dir\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.910764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.910734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-model-cache\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.910896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.910775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-home\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.910896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.910798    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19124252-c993-4fb7-a514-1fa16ba8a3af-tls-certs\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.911129 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.911103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64wd4\" (UniqueName: \"kubernetes.io/projected/19124252-c993-4fb7-a514-1fa16ba8a3af-kube-api-access-64wd4\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.911129 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.911122    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-tmp-dir\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.911302 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.911197    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-home\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.911407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.911382    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-model-cache\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.913030 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.913014    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-dshm\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.913281 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.913265    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19124252-c993-4fb7-a514-1fa16ba8a3af-tls-certs\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:54.919705 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:54.919688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64wd4\" (UniqueName: \"kubernetes.io/projected/19124252-c993-4fb7-a514-1fa16ba8a3af-kube-api-access-64wd4\") pod \"auth-test-model-7be222-kserve-c49dff79f-m7kqv\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") " pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:55.080948 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:55.080924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:14:55.351970 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:55.351907    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"]
Sep 08 15:14:55.355256 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:14:55.355231    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19124252_c993_4fb7_a514_1fa16ba8a3af.slice/crio-14001d828f238a3ca34fd79fe448a25dd65277585f722a92894da024e94a8bd2 WatchSource:0}: Error finding container 14001d828f238a3ca34fd79fe448a25dd65277585f722a92894da024e94a8bd2: Status 404 returned error can't find the container with id 14001d828f238a3ca34fd79fe448a25dd65277585f722a92894da024e94a8bd2
Sep 08 15:14:55.878992 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:55.878956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv" event={"ID":"19124252-c993-4fb7-a514-1fa16ba8a3af","Type":"ContainerStarted","Data":"eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70"}
Sep 08 15:14:55.878992 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:55.879005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv" event={"ID":"19124252-c993-4fb7-a514-1fa16ba8a3af","Type":"ContainerStarted","Data":"14001d828f238a3ca34fd79fe448a25dd65277585f722a92894da024e94a8bd2"}
Sep 08 15:14:55.895596 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:14:55.895547    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv" podStartSLOduration=1.8955109060000002 podStartE2EDuration="1.895510906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:14:55.893919531 +0000 UTC m=+1174.545343843" watchObservedRunningTime="2026-09-08 15:14:55.895510906 +0000 UTC m=+1174.546935214"
Sep 08 15:15:00.143466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.143425    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"]
Sep 08 15:15:00.146978 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.146947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"]
Sep 08 15:15:00.147115 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.147039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"
Sep 08 15:15:00.149201 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.149177    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b8vrf\""
Sep 08 15:15:00.150437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.150402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"
Sep 08 15:15:00.156315 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.156293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"]
Sep 08 15:15:00.158968 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.158942    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"]
Sep 08 15:15:00.163515 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.163496    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"
Sep 08 15:15:00.170518 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.170502    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"
Sep 08 15:15:00.449012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.448782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"]
Sep 08 15:15:00.452415 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:15:00.452389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e61371_6a4b_48ca_96bd_95a5aabe7886.slice/crio-6a8a1e5c02ac31277f8bca4bac2c0a78aef2e885ce1715b04cb81c9bff08bc1e WatchSource:0}: Error finding container 6a8a1e5c02ac31277f8bca4bac2c0a78aef2e885ce1715b04cb81c9bff08bc1e: Status 404 returned error can't find the container with id 6a8a1e5c02ac31277f8bca4bac2c0a78aef2e885ce1715b04cb81c9bff08bc1e
Sep 08 15:15:00.454794 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.454763    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:15:00.467310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.467290    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"]
Sep 08 15:15:00.469480 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:15:00.469457    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fbd6c9_1a51_4b11_bf42_c3f4d8e618c5.slice/crio-6534f0b52386028102cf36e4a90d16d7c869aa2d64379965968c4eca6f27d136 WatchSource:0}: Error finding container 6534f0b52386028102cf36e4a90d16d7c869aa2d64379965968c4eca6f27d136: Status 404 returned error can't find the container with id 6534f0b52386028102cf36e4a90d16d7c869aa2d64379965968c4eca6f27d136
Sep 08 15:15:00.905352 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.905317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66" event={"ID":"b9e61371-6a4b-48ca-96bd-95a5aabe7886","Type":"ContainerStarted","Data":"6a8a1e5c02ac31277f8bca4bac2c0a78aef2e885ce1715b04cb81c9bff08bc1e"}
Sep 08 15:15:00.906556 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:00.906511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl" event={"ID":"71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5","Type":"ContainerStarted","Data":"6534f0b52386028102cf36e4a90d16d7c869aa2d64379965968c4eca6f27d136"}
Sep 08 15:15:02.919466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:02.919440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl" event={"ID":"71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5","Type":"ContainerStarted","Data":"ed5e3c43c96ac4f2ae129eb3dfe95314018188ffa09a6e9c5cc8d92dffad9460"}
Sep 08 15:15:02.937122 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:02.937076    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl" podStartSLOduration=0.697029828 podStartE2EDuration="2.937064116s" totalImagesPullingTime="2.240034288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:15:02.935483779 +0000 UTC m=+1181.586908089" watchObservedRunningTime="2026-09-08 15:15:02.937064116 +0000 UTC m=+1181.588488476"
Sep 08 15:15:03.925898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:03.925825    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9e61371-6a4b-48ca-96bd-95a5aabe7886" containerID="6c662d487d6a3feb4b33207532d289acd8b2d2b962fde01aaf20e79707d49946" exitCode=0
Sep 08 15:15:03.926293 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:03.925914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66" event={"ID":"b9e61371-6a4b-48ca-96bd-95a5aabe7886","Type":"ContainerDied","Data":"6c662d487d6a3feb4b33207532d289acd8b2d2b962fde01aaf20e79707d49946"}
Sep 08 15:15:03.927547 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:03.927510    2578 generic.go:356] "Generic (PLEG): container finished" podID="71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5" containerID="ed5e3c43c96ac4f2ae129eb3dfe95314018188ffa09a6e9c5cc8d92dffad9460" exitCode=0
Sep 08 15:15:03.927635 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:03.927570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl" event={"ID":"71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5","Type":"ContainerDied","Data":"ed5e3c43c96ac4f2ae129eb3dfe95314018188ffa09a6e9c5cc8d92dffad9460"}
Sep 08 15:15:05.081704 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.081682    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:15:05.082068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.081729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:15:05.094939 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.094914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:15:05.095515 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.095495    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"
Sep 08 15:15:05.099413 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.099394    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"
Sep 08 15:15:05.939012 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.938987    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl"
Sep 08 15:15:05.939164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.939030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814675-xx6pl" event={"ID":"71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5","Type":"ContainerDied","Data":"6534f0b52386028102cf36e4a90d16d7c869aa2d64379965968c4eca6f27d136"}
Sep 08 15:15:05.939164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.939086    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6534f0b52386028102cf36e4a90d16d7c869aa2d64379965968c4eca6f27d136"
Sep 08 15:15:05.940453 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.940434    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"
Sep 08 15:15:05.940579 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.940480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66" event={"ID":"b9e61371-6a4b-48ca-96bd-95a5aabe7886","Type":"ContainerDied","Data":"6a8a1e5c02ac31277f8bca4bac2c0a78aef2e885ce1715b04cb81c9bff08bc1e"}
Sep 08 15:15:05.940579 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.940547    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8a1e5c02ac31277f8bca4bac2c0a78aef2e885ce1715b04cb81c9bff08bc1e"
Sep 08 15:15:05.952506 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:05.952477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:15:10.031180 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:10.031142    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:10.345760 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:10.345655    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:10.974810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:10.974780    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:13.179610 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:13.179582    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:13.264257 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:13.264224    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:14.466372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:14.466331    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:14.570507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:14.570475    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:14.677414 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:14.677380    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:15.114728 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.114698    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:15.214893 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.214862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:15.269825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.269799    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"]
Sep 08 15:15:15.270164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.270152    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e61371-6a4b-48ca-96bd-95a5aabe7886" containerName="cleanup"
Sep 08 15:15:15.270210 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.270166    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e61371-6a4b-48ca-96bd-95a5aabe7886" containerName="cleanup"
Sep 08 15:15:15.270210 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.270180    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5" containerName="cleanup"
Sep 08 15:15:15.270210 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.270185    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5" containerName="cleanup"
Sep 08 15:15:15.270316 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.270259    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5" containerName="cleanup"
Sep 08 15:15:15.270316 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.270270    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e61371-6a4b-48ca-96bd-95a5aabe7886" containerName="cleanup"
Sep 08 15:15:15.275599 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.274297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.279004 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.278975    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-48dbdcd3-dockercfg-7cbzt\""
Sep 08 15:15:15.279128 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.279063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-48dbdcd3\""
Sep 08 15:15:15.284988 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.284969    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"]
Sep 08 15:15:15.368748 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.368684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2pn4\" (UniqueName: \"kubernetes.io/projected/0279b64d-976d-431e-94aa-5e1388263909-kube-api-access-p2pn4\") pod \"payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") " pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.368748 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.368726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0279b64d-976d-431e-94aa-5e1388263909-plugins-config-volume\") pod \"payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") " pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.469254 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.469229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0279b64d-976d-431e-94aa-5e1388263909-plugins-config-volume\") pod \"payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") " pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.469600 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.469310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2pn4\" (UniqueName: \"kubernetes.io/projected/0279b64d-976d-431e-94aa-5e1388263909-kube-api-access-p2pn4\") pod \"payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") " pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.469930 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.469912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0279b64d-976d-431e-94aa-5e1388263909-plugins-config-volume\") pod \"payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") " pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.479692 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.479673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2pn4\" (UniqueName: \"kubernetes.io/projected/0279b64d-976d-431e-94aa-5e1388263909-kube-api-access-p2pn4\") pod \"payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") " pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.589720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.589693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:15.861332 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.861305    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"]
Sep 08 15:15:15.863326 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:15:15.863304    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0279b64d_976d_431e_94aa_5e1388263909.slice/crio-c69027eb6d47972a31db8f44480242f79f5242d450735b16ce757df04abfc53f WatchSource:0}: Error finding container c69027eb6d47972a31db8f44480242f79f5242d450735b16ce757df04abfc53f: Status 404 returned error can't find the container with id c69027eb6d47972a31db8f44480242f79f5242d450735b16ce757df04abfc53f
Sep 08 15:15:15.993413 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.993380    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs" event={"ID":"0279b64d-976d-431e-94aa-5e1388263909","Type":"ContainerStarted","Data":"f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5"}
Sep 08 15:15:15.993637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.993439    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs" event={"ID":"0279b64d-976d-431e-94aa-5e1388263909","Type":"ContainerStarted","Data":"c69027eb6d47972a31db8f44480242f79f5242d450735b16ce757df04abfc53f"}
Sep 08 15:15:15.993637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:15.993513    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:16.009514 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:16.009459    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs" podStartSLOduration=1.009441542 podStartE2EDuration="1.009441542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:15:16.008947822 +0000 UTC m=+1194.660372128" watchObservedRunningTime="2026-09-08 15:15:16.009441542 +0000 UTC m=+1194.660865852"
Sep 08 15:15:21.944994 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:21.944967    2578 scope.go:117] "RemoveContainer" containerID="da4ab75a50791b95c170d797de318e1b4ec68d5400de0b2f7ecf892d36e83da7"
Sep 08 15:15:21.953851 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:21.953835    2578 scope.go:117] "RemoveContainer" containerID="40111f3b1d85dc697f0902c48cf60b18209e69423df7f11eecd7ded9040e1353"
Sep 08 15:15:24.958329 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:24.958296    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:27.000634 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:27.000596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:15:38.126943 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:38.126912    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:46.016929 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:46.016893    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:51.468309 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:51.468249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:51.533644 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:51.533617    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:15:58.023098 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:15:58.023056    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:07.503412 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:07.503382    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:22.751116 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:22.751080    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:38.053397 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.053361    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:38.080469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.080443    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"]
Sep 08 15:16:38.080673 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.080650    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs" podUID="0279b64d-976d-431e-94aa-5e1388263909" containerName="payload-processing" containerID="cri-o://f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5" gracePeriod=30
Sep 08 15:16:38.155952 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.155916    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:38.256462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.256417    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:38.339298 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.339277    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:16:38.357424 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.357395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:38.409464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.409436    2578 generic.go:356] "Generic (PLEG): container finished" podID="0279b64d-976d-431e-94aa-5e1388263909" containerID="f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5" exitCode=0
Sep 08 15:16:38.409627 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.409485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"
Sep 08 15:16:38.409627 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.409517    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs" event={"ID":"0279b64d-976d-431e-94aa-5e1388263909","Type":"ContainerDied","Data":"f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5"}
Sep 08 15:16:38.409627 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.409617    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs" event={"ID":"0279b64d-976d-431e-94aa-5e1388263909","Type":"ContainerDied","Data":"c69027eb6d47972a31db8f44480242f79f5242d450735b16ce757df04abfc53f"}
Sep 08 15:16:38.409760 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.409644    2578 scope.go:117] "RemoveContainer" containerID="f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5"
Sep 08 15:16:38.417080 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.417062    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2pn4\" (UniqueName: \"kubernetes.io/projected/0279b64d-976d-431e-94aa-5e1388263909-kube-api-access-p2pn4\") pod \"0279b64d-976d-431e-94aa-5e1388263909\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") "
Sep 08 15:16:38.417204 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.417100    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0279b64d-976d-431e-94aa-5e1388263909-plugins-config-volume\") pod \"0279b64d-976d-431e-94aa-5e1388263909\" (UID: \"0279b64d-976d-431e-94aa-5e1388263909\") "
Sep 08 15:16:38.417613 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.417590    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0279b64d-976d-431e-94aa-5e1388263909-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0279b64d-976d-431e-94aa-5e1388263909" (UID: "0279b64d-976d-431e-94aa-5e1388263909"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:16:38.418182 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.418162    2578 scope.go:117] "RemoveContainer" containerID="f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5"
Sep 08 15:16:38.418478 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:16:38.418455    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5\": container with ID starting with f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5 not found: ID does not exist" containerID="f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5"
Sep 08 15:16:38.418556 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.418492    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5"} err="failed to get container status \"f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5\": rpc error: code = NotFound desc = could not find container \"f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5\": container with ID starting with f58f3e5e374d9a0a8a5c62e497efabb3860667c738dffbb72b52cccf421448b5 not found: ID does not exist"
Sep 08 15:16:38.419123 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.419102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0279b64d-976d-431e-94aa-5e1388263909-kube-api-access-p2pn4" (OuterVolumeSpecName: "kube-api-access-p2pn4") pod "0279b64d-976d-431e-94aa-5e1388263909" (UID: "0279b64d-976d-431e-94aa-5e1388263909"). InnerVolumeSpecName "kube-api-access-p2pn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:16:38.518129 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.518101    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2pn4\" (UniqueName: \"kubernetes.io/projected/0279b64d-976d-431e-94aa-5e1388263909-kube-api-access-p2pn4\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:16:38.518129 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.518120    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0279b64d-976d-431e-94aa-5e1388263909-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:16:38.731362 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.731342    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"]
Sep 08 15:16:38.734762 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:38.734742    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-48dbdcd3-68fc57b745-nmvcs"]
Sep 08 15:16:39.832244 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:39.832213    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0279b64d-976d-431e-94aa-5e1388263909" path="/var/lib/kubelet/pods/0279b64d-976d-431e-94aa-5e1388263909/volumes"
Sep 08 15:16:41.351962 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:41.351923    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:46.376061 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:46.376024    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:16:47.966174 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:16:47.966133    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:00.528310 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:00.528272    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"]
Sep 08 15:17:00.549955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:00.549917    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:01.806417 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:01.806384    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"]
Sep 08 15:17:01.857119 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:01.857092    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:15.549677 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.549617    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv" podUID="25c31545-492b-46fd-83b0-f667706bd5cc" containerName="main" containerID="cri-o://0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e" gracePeriod=45
Sep 08 15:17:15.799158 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.799137    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:17:15.899290 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-tmp-dir\") pod \"25c31545-492b-46fd-83b0-f667706bd5cc\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") "
Sep 08 15:17:15.899290 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899255    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-home\") pod \"25c31545-492b-46fd-83b0-f667706bd5cc\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") "
Sep 08 15:17:15.899290 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899280    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnj2n\" (UniqueName: \"kubernetes.io/projected/25c31545-492b-46fd-83b0-f667706bd5cc-kube-api-access-rnj2n\") pod \"25c31545-492b-46fd-83b0-f667706bd5cc\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") "
Sep 08 15:17:15.899574 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c31545-492b-46fd-83b0-f667706bd5cc-tls-certs\") pod \"25c31545-492b-46fd-83b0-f667706bd5cc\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") "
Sep 08 15:17:15.899574 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "25c31545-492b-46fd-83b0-f667706bd5cc" (UID: "25c31545-492b-46fd-83b0-f667706bd5cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:15.899574 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-home" (OuterVolumeSpecName: "home") pod "25c31545-492b-46fd-83b0-f667706bd5cc" (UID: "25c31545-492b-46fd-83b0-f667706bd5cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:15.899733 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899609    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:15.899733 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.899622    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:15.901671 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.901650    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25c31545-492b-46fd-83b0-f667706bd5cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "25c31545-492b-46fd-83b0-f667706bd5cc" (UID: "25c31545-492b-46fd-83b0-f667706bd5cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:17:15.901778 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.901762    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25c31545-492b-46fd-83b0-f667706bd5cc-kube-api-access-rnj2n" (OuterVolumeSpecName: "kube-api-access-rnj2n") pod "25c31545-492b-46fd-83b0-f667706bd5cc" (UID: "25c31545-492b-46fd-83b0-f667706bd5cc"). InnerVolumeSpecName "kube-api-access-rnj2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:17:15.999921 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.999887    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-dshm\") pod \"25c31545-492b-46fd-83b0-f667706bd5cc\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") "
Sep 08 15:17:15.999921 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:15.999928    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-model-cache\") pod \"25c31545-492b-46fd-83b0-f667706bd5cc\" (UID: \"25c31545-492b-46fd-83b0-f667706bd5cc\") "
Sep 08 15:17:16.000149 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.000136    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnj2n\" (UniqueName: \"kubernetes.io/projected/25c31545-492b-46fd-83b0-f667706bd5cc-kube-api-access-rnj2n\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:16.000188 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.000152    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/25c31545-492b-46fd-83b0-f667706bd5cc-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:16.000221 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.000191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "25c31545-492b-46fd-83b0-f667706bd5cc" (UID: "25c31545-492b-46fd-83b0-f667706bd5cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:16.001874 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.001856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-dshm" (OuterVolumeSpecName: "dshm") pod "25c31545-492b-46fd-83b0-f667706bd5cc" (UID: "25c31545-492b-46fd-83b0-f667706bd5cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:16.100823 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.100795    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:16.100897 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.100826    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/25c31545-492b-46fd-83b0-f667706bd5cc-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:16.198266 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198212    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"]
Sep 08 15:17:16.198607 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198594    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0279b64d-976d-431e-94aa-5e1388263909" containerName="payload-processing"
Sep 08 15:17:16.198657 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198609    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0279b64d-976d-431e-94aa-5e1388263909" containerName="payload-processing"
Sep 08 15:17:16.198657 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198629    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25c31545-492b-46fd-83b0-f667706bd5cc" containerName="main"
Sep 08 15:17:16.198657 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198634    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="25c31545-492b-46fd-83b0-f667706bd5cc" containerName="main"
Sep 08 15:17:16.198757 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198697    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="25c31545-492b-46fd-83b0-f667706bd5cc" containerName="main"
Sep 08 15:17:16.198757 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.198705    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0279b64d-976d-431e-94aa-5e1388263909" containerName="payload-processing"
Sep 08 15:17:16.201744 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.201728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.206761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.206742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-ffa941\"/\"sub-test-model-ffa941-kserve-self-signed-certs\""
Sep 08 15:17:16.225926 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.225900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"]
Sep 08 15:17:16.240363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.240342    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:16.302812 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.302789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-tmp-dir\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.302914 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.302813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c82a4c-a088-4105-a647-eac435db3d4a-tls-certs\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.302914 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.302839    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qchv\" (UniqueName: \"kubernetes.io/projected/e4c82a4c-a088-4105-a647-eac435db3d4a-kube-api-access-9qchv\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.303039 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.302978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-dshm\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.303039 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.303013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-model-cache\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.303155 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.303060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-home\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.403702 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.403679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qchv\" (UniqueName: \"kubernetes.io/projected/e4c82a4c-a088-4105-a647-eac435db3d4a-kube-api-access-9qchv\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.403810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.403729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-dshm\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.403810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.403746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-model-cache\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.403810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.403772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-home\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.403810 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.403800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-tmp-dir\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.404032 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.403822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c82a4c-a088-4105-a647-eac435db3d4a-tls-certs\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.404211 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.404189    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-home\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.404273 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.404207    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-tmp-dir\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.404273 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.404248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-model-cache\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.406187 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.406169    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-dshm\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.406356 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.406340    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c82a4c-a088-4105-a647-eac435db3d4a-tls-certs\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.412203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.412180    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qchv\" (UniqueName: \"kubernetes.io/projected/e4c82a4c-a088-4105-a647-eac435db3d4a-kube-api-access-9qchv\") pod \"sub-test-model-ffa941-kserve-67c85b547c-7tgpz\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") " pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.511663 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.511637    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:16.613822 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.613791    2578 generic.go:356] "Generic (PLEG): container finished" podID="25c31545-492b-46fd-83b0-f667706bd5cc" containerID="0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e" exitCode=0
Sep 08 15:17:16.614208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.613841    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"
Sep 08 15:17:16.614208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.613848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv" event={"ID":"25c31545-492b-46fd-83b0-f667706bd5cc","Type":"ContainerDied","Data":"0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e"}
Sep 08 15:17:16.614208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.613880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv" event={"ID":"25c31545-492b-46fd-83b0-f667706bd5cc","Type":"ContainerDied","Data":"15d73a595c3545c1faaa438864623a0203487f9536e212cbd940f4e8a6e0289c"}
Sep 08 15:17:16.614208 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.613896    2578 scope.go:117] "RemoveContainer" containerID="0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e"
Sep 08 15:17:16.622630 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.622610    2578 scope.go:117] "RemoveContainer" containerID="0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e"
Sep 08 15:17:16.622890 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:17:16.622870    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e\": container with ID starting with 0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e not found: ID does not exist" containerID="0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e"
Sep 08 15:17:16.622955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.622897    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e"} err="failed to get container status \"0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e\": rpc error: code = NotFound desc = could not find container \"0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e\": container with ID starting with 0090b341e5cc01b108b30f1a9868652329b69651643febadfd3578978b294d7e not found: ID does not exist"
Sep 08 15:17:16.635212 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.635190    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"]
Sep 08 15:17:16.639308 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.639288    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/auth-test-model-ffa941-kserve-6d7bb759d7-fgzfv"]
Sep 08 15:17:16.783655 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.783619    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"]
Sep 08 15:17:16.785881 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:17:16.785861    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4c82a4c_a088_4105_a647_eac435db3d4a.slice/crio-9026d826fbcf49d0204965bb940e7e9f4a8da29cf7bb84a794bb2d4213fafde5 WatchSource:0}: Error finding container 9026d826fbcf49d0204965bb940e7e9f4a8da29cf7bb84a794bb2d4213fafde5: Status 404 returned error can't find the container with id 9026d826fbcf49d0204965bb940e7e9f4a8da29cf7bb84a794bb2d4213fafde5
Sep 08 15:17:16.825851 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:16.825811    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv" podUID="19124252-c993-4fb7-a514-1fa16ba8a3af" containerName="main" containerID="cri-o://eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70" gracePeriod=45
Sep 08 15:17:17.072003 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.071982    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:17:17.108886 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.108865    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19124252-c993-4fb7-a514-1fa16ba8a3af-tls-certs\") pod \"19124252-c993-4fb7-a514-1fa16ba8a3af\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") "
Sep 08 15:17:17.109016 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.108972    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-tmp-dir\") pod \"19124252-c993-4fb7-a514-1fa16ba8a3af\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") "
Sep 08 15:17:17.109016 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.109002    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-dshm\") pod \"19124252-c993-4fb7-a514-1fa16ba8a3af\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") "
Sep 08 15:17:17.109144 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.109034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64wd4\" (UniqueName: \"kubernetes.io/projected/19124252-c993-4fb7-a514-1fa16ba8a3af-kube-api-access-64wd4\") pod \"19124252-c993-4fb7-a514-1fa16ba8a3af\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") "
Sep 08 15:17:17.109265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.109236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "19124252-c993-4fb7-a514-1fa16ba8a3af" (UID: "19124252-c993-4fb7-a514-1fa16ba8a3af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:17.109417 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.109351    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:17.111175 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.111147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-dshm" (OuterVolumeSpecName: "dshm") pod "19124252-c993-4fb7-a514-1fa16ba8a3af" (UID: "19124252-c993-4fb7-a514-1fa16ba8a3af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:17.111272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.111219    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19124252-c993-4fb7-a514-1fa16ba8a3af-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "19124252-c993-4fb7-a514-1fa16ba8a3af" (UID: "19124252-c993-4fb7-a514-1fa16ba8a3af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:17:17.111316 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.111270    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19124252-c993-4fb7-a514-1fa16ba8a3af-kube-api-access-64wd4" (OuterVolumeSpecName: "kube-api-access-64wd4") pod "19124252-c993-4fb7-a514-1fa16ba8a3af" (UID: "19124252-c993-4fb7-a514-1fa16ba8a3af"). InnerVolumeSpecName "kube-api-access-64wd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:17:17.210491 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-model-cache\") pod \"19124252-c993-4fb7-a514-1fa16ba8a3af\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") "
Sep 08 15:17:17.210611 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210497    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-home\") pod \"19124252-c993-4fb7-a514-1fa16ba8a3af\" (UID: \"19124252-c993-4fb7-a514-1fa16ba8a3af\") "
Sep 08 15:17:17.210694 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210681    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:17.210738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210702    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64wd4\" (UniqueName: \"kubernetes.io/projected/19124252-c993-4fb7-a514-1fa16ba8a3af-kube-api-access-64wd4\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:17.210738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210712    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19124252-c993-4fb7-a514-1fa16ba8a3af-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:17.210738 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210677    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-model-cache" (OuterVolumeSpecName: "model-cache") pod "19124252-c993-4fb7-a514-1fa16ba8a3af" (UID: "19124252-c993-4fb7-a514-1fa16ba8a3af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:17.210838 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.210759    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-home" (OuterVolumeSpecName: "home") pod "19124252-c993-4fb7-a514-1fa16ba8a3af" (UID: "19124252-c993-4fb7-a514-1fa16ba8a3af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:17:17.311697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.311644    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:17.311697 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.311664    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19124252-c993-4fb7-a514-1fa16ba8a3af-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:17:17.621137 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.621109    2578 generic.go:356] "Generic (PLEG): container finished" podID="19124252-c993-4fb7-a514-1fa16ba8a3af" containerID="eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70" exitCode=0
Sep 08 15:17:17.621507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.621156    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"
Sep 08 15:17:17.621507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.621195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv" event={"ID":"19124252-c993-4fb7-a514-1fa16ba8a3af","Type":"ContainerDied","Data":"eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70"}
Sep 08 15:17:17.621507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.621246    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv" event={"ID":"19124252-c993-4fb7-a514-1fa16ba8a3af","Type":"ContainerDied","Data":"14001d828f238a3ca34fd79fe448a25dd65277585f722a92894da024e94a8bd2"}
Sep 08 15:17:17.621507 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.621273    2578 scope.go:117] "RemoveContainer" containerID="eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70"
Sep 08 15:17:17.623174 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.623148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz" event={"ID":"e4c82a4c-a088-4105-a647-eac435db3d4a","Type":"ContainerStarted","Data":"a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad"}
Sep 08 15:17:17.623261 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.623188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz" event={"ID":"e4c82a4c-a088-4105-a647-eac435db3d4a","Type":"ContainerStarted","Data":"9026d826fbcf49d0204965bb940e7e9f4a8da29cf7bb84a794bb2d4213fafde5"}
Sep 08 15:17:17.630487 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.630470    2578 scope.go:117] "RemoveContainer" containerID="eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70"
Sep 08 15:17:17.630794 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:17:17.630777    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70\": container with ID starting with eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70 not found: ID does not exist" containerID="eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70"
Sep 08 15:17:17.630854 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.630802    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70"} err="failed to get container status \"eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70\": rpc error: code = NotFound desc = could not find container \"eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70\": container with ID starting with eeafa168a0c02bde9e6c8c9490176d01a609c22ff9a8c38edf2822851f485f70 not found: ID does not exist"
Sep 08 15:17:17.642558 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.642500    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz" podStartSLOduration=1.642488266 podStartE2EDuration="1.642488266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:17.641413528 +0000 UTC m=+1316.292837838" watchObservedRunningTime="2026-09-08 15:17:17.642488266 +0000 UTC m=+1316.293912576"
Sep 08 15:17:17.656723 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.656705    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"]
Sep 08 15:17:17.660679 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.660659    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/auth-test-model-7be222-kserve-c49dff79f-m7kqv"]
Sep 08 15:17:17.831095 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.831040    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19124252-c993-4fb7-a514-1fa16ba8a3af" path="/var/lib/kubelet/pods/19124252-c993-4fb7-a514-1fa16ba8a3af/volumes"
Sep 08 15:17:17.831406 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:17.831392    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25c31545-492b-46fd-83b0-f667706bd5cc" path="/var/lib/kubelet/pods/25c31545-492b-46fd-83b0-f667706bd5cc/volumes"
Sep 08 15:17:22.507792 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:22.507739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:23.608935 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:23.608901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:26.512490 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:26.512454    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:26.512942 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:26.512506    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:26.525568 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:26.525514    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:26.681570 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:26.681546    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:17:29.427844 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:29.427806    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:29.496019 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:29.495971    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:30.165671 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:30.165643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:30.996821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:30.996782    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:31.096381 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.096355    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:31.710464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.710435    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:31.811039 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.811013    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:31.911283 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.911236    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:31.994255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.994182    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:31.995564 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.995541    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"]
Sep 08 15:17:31.995972 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.995952    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19124252-c993-4fb7-a514-1fa16ba8a3af" containerName="main"
Sep 08 15:17:31.995972 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.995974    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="19124252-c993-4fb7-a514-1fa16ba8a3af" containerName="main"
Sep 08 15:17:31.996143 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:31.996118    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="19124252-c993-4fb7-a514-1fa16ba8a3af" containerName="main"
Sep 08 15:17:32.000655 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.000639    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.008117 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.008100    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"kube-root-ca.crt\""
Sep 08 15:17:32.009203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.009182    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"sub-test-model-7be222-kserve-self-signed-certs\""
Sep 08 15:17:32.009203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.009197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"default-dockercfg-zm699\""
Sep 08 15:17:32.009382 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.009197    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"openshift-service-ca.crt\""
Sep 08 15:17:32.017490 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.017465    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"]
Sep 08 15:17:32.019223 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.019199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9nnc\" (UniqueName: \"kubernetes.io/projected/c4e52d79-75e1-4fea-b886-20b40734d8a5-kube-api-access-z9nnc\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.019340 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.019242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-tmp-dir\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.019340 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.019279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-model-cache\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.019470 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.019348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-dshm\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.019470 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.019432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e52d79-75e1-4fea-b886-20b40734d8a5-tls-certs\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.019470 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.019454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-home\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.067632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.067606    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:32.120005 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.119982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e52d79-75e1-4fea-b886-20b40734d8a5-tls-certs\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120009    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-home\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120041    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9nnc\" (UniqueName: \"kubernetes.io/projected/c4e52d79-75e1-4fea-b886-20b40734d8a5-kube-api-access-z9nnc\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-tmp-dir\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120114 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120095    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-model-cache\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120286 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-dshm\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120436 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120421    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-tmp-dir\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120513 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120495    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-model-cache\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.120650 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.120634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-home\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.122428 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.122405    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-dshm\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.122744 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.122726    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e52d79-75e1-4fea-b886-20b40734d8a5-tls-certs\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.128452 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.128425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9nnc\" (UniqueName: \"kubernetes.io/projected/c4e52d79-75e1-4fea-b886-20b40734d8a5-kube-api-access-z9nnc\") pod \"sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") " pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.310794 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.310773    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:32.583821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.583796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"]
Sep 08 15:17:32.585574 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:17:32.585548    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4e52d79_75e1_4fea_b886_20b40734d8a5.slice/crio-23b2f6bbbce488e5878e0d744bd27fa56d7c9c5aae0804d21fd531bc3de73462 WatchSource:0}: Error finding container 23b2f6bbbce488e5878e0d744bd27fa56d7c9c5aae0804d21fd531bc3de73462: Status 404 returned error can't find the container with id 23b2f6bbbce488e5878e0d744bd27fa56d7c9c5aae0804d21fd531bc3de73462
Sep 08 15:17:32.702161 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.702138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd" event={"ID":"c4e52d79-75e1-4fea-b886-20b40734d8a5","Type":"ContainerStarted","Data":"6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb"}
Sep 08 15:17:32.702292 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.702171    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd" event={"ID":"c4e52d79-75e1-4fea-b886-20b40734d8a5","Type":"ContainerStarted","Data":"23b2f6bbbce488e5878e0d744bd27fa56d7c9c5aae0804d21fd531bc3de73462"}
Sep 08 15:17:32.720145 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:32.720102    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd" podStartSLOduration=1.720087567 podStartE2EDuration="1.720087567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:17:32.717556727 +0000 UTC m=+1331.368981035" watchObservedRunningTime="2026-09-08 15:17:32.720087567 +0000 UTC m=+1331.371511876"
Sep 08 15:17:33.256857 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:33.256821    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:39.851473 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:39.851437    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:39.949940 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:39.949915    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:40.524841 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:40.524804    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:42.311038 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:42.311000    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:42.311401 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:42.311264    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:42.323542 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:42.323499    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:42.764487 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:42.764463    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:17:46.381691 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:46.381649    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:47.052681 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:47.052654    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:47.119576 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:47.119548    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:57.651583 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:57.651508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:17:57.903183 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:17:57.903107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:18.957437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:18.957403    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:22.002148 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:22.002118    2578 scope.go:117] "RemoveContainer" containerID="67cd04feda6fe0bbbd643b4765269d5eef218f41992e6b4de815b7a7bb92fbb9"
Sep 08 15:18:24.294854 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:24.294812    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"]
Sep 08 15:18:24.320062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:24.320033    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:25.281420 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:25.281387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"]
Sep 08 15:18:26.575440 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:26.575395    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:26.662674 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:26.662643    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:26.773620 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:26.773584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:26.934285 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:26.934214    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:29.851980 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:29.851946    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:29.944934 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:29.944905    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:31.131667 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.131637    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:31.232784 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.232755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:31.332314 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.332241    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:31.616024 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.615997    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:31.686939 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.686913    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:31.736391 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.736365    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"]
Sep 08 15:18:31.741399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.741384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:31.743900 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.743880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3ccde3\""
Sep 08 15:18:31.749564 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.749545    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"]
Sep 08 15:18:31.882776 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.882709    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fdc23-d569-4583-acb4-5f0bd8e80173-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:31.882776 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.882740    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67h25\" (UniqueName: \"kubernetes.io/projected/c35fdc23-d569-4583-acb4-5f0bd8e80173-kube-api-access-67h25\") pod \"payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:31.983643 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.983609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fdc23-d569-4583-acb4-5f0bd8e80173-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:31.983643 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.983644    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67h25\" (UniqueName: \"kubernetes.io/projected/c35fdc23-d569-4583-acb4-5f0bd8e80173-kube-api-access-67h25\") pod \"payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:31.984289 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.984270    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fdc23-d569-4583-acb4-5f0bd8e80173-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:31.992142 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:31.992115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67h25\" (UniqueName: \"kubernetes.io/projected/c35fdc23-d569-4583-acb4-5f0bd8e80173-kube-api-access-67h25\") pod \"payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") " pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:32.053891 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:32.053868    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:32.325441 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:32.325417    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"]
Sep 08 15:18:32.328445 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:18:32.328418    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35fdc23_d569_4583_acb4_5f0bd8e80173.slice/crio-587762e320a1f52b52a59103041371feaee06d90035025a162167098c0bc28b2 WatchSource:0}: Error finding container 587762e320a1f52b52a59103041371feaee06d90035025a162167098c0bc28b2: Status 404 returned error can't find the container with id 587762e320a1f52b52a59103041371feaee06d90035025a162167098c0bc28b2
Sep 08 15:18:33.001268 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:33.001236    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq" event={"ID":"c35fdc23-d569-4583-acb4-5f0bd8e80173","Type":"ContainerStarted","Data":"dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7"}
Sep 08 15:18:33.001440 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:33.001283    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq" event={"ID":"c35fdc23-d569-4583-acb4-5f0bd8e80173","Type":"ContainerStarted","Data":"587762e320a1f52b52a59103041371feaee06d90035025a162167098c0bc28b2"}
Sep 08 15:18:33.001440 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:33.001352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:33.018088 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:33.018041    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq" podStartSLOduration=2.018029098 podStartE2EDuration="2.018029098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:18:33.015803045 +0000 UTC m=+1391.667227367" watchObservedRunningTime="2026-09-08 15:18:33.018029098 +0000 UTC m=+1391.669453407"
Sep 08 15:18:36.436066 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:36.436030    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:39.315582 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.315493    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz" podUID="e4c82a4c-a088-4105-a647-eac435db3d4a" containerName="main" containerID="cri-o://a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad" gracePeriod=45
Sep 08 15:18:39.574603 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.574514    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:18:39.643814 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.643791    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qchv\" (UniqueName: \"kubernetes.io/projected/e4c82a4c-a088-4105-a647-eac435db3d4a-kube-api-access-9qchv\") pod \"e4c82a4c-a088-4105-a647-eac435db3d4a\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") "
Sep 08 15:18:39.643953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.643822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c82a4c-a088-4105-a647-eac435db3d4a-tls-certs\") pod \"e4c82a4c-a088-4105-a647-eac435db3d4a\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") "
Sep 08 15:18:39.643953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.643896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-dshm\") pod \"e4c82a4c-a088-4105-a647-eac435db3d4a\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") "
Sep 08 15:18:39.643953 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.643919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-model-cache\") pod \"e4c82a4c-a088-4105-a647-eac435db3d4a\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") "
Sep 08 15:18:39.644092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.643956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-tmp-dir\") pod \"e4c82a4c-a088-4105-a647-eac435db3d4a\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") "
Sep 08 15:18:39.644092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.644012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-home\") pod \"e4c82a4c-a088-4105-a647-eac435db3d4a\" (UID: \"e4c82a4c-a088-4105-a647-eac435db3d4a\") "
Sep 08 15:18:39.644224 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.644202    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4c82a4c-a088-4105-a647-eac435db3d4a" (UID: "e4c82a4c-a088-4105-a647-eac435db3d4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:39.644295 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.644248    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4c82a4c-a088-4105-a647-eac435db3d4a" (UID: "e4c82a4c-a088-4105-a647-eac435db3d4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:39.644357 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.644310    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-home" (OuterVolumeSpecName: "home") pod "e4c82a4c-a088-4105-a647-eac435db3d4a" (UID: "e4c82a4c-a088-4105-a647-eac435db3d4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:39.644357 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.644347    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:39.646101 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.646079    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4c82a4c-a088-4105-a647-eac435db3d4a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4c82a4c-a088-4105-a647-eac435db3d4a" (UID: "e4c82a4c-a088-4105-a647-eac435db3d4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:39.646239 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.646131    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-dshm" (OuterVolumeSpecName: "dshm") pod "e4c82a4c-a088-4105-a647-eac435db3d4a" (UID: "e4c82a4c-a088-4105-a647-eac435db3d4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:39.646239 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.646145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c82a4c-a088-4105-a647-eac435db3d4a-kube-api-access-9qchv" (OuterVolumeSpecName: "kube-api-access-9qchv") pod "e4c82a4c-a088-4105-a647-eac435db3d4a" (UID: "e4c82a4c-a088-4105-a647-eac435db3d4a"). InnerVolumeSpecName "kube-api-access-9qchv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:39.745563 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.745543    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:39.745670 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.745580    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:39.745670 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.745591    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4c82a4c-a088-4105-a647-eac435db3d4a-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:39.745670 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.745599    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qchv\" (UniqueName: \"kubernetes.io/projected/e4c82a4c-a088-4105-a647-eac435db3d4a-kube-api-access-9qchv\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:39.745670 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:39.745608    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4c82a4c-a088-4105-a647-eac435db3d4a-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:40.037018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.036991    2578 generic.go:356] "Generic (PLEG): container finished" podID="e4c82a4c-a088-4105-a647-eac435db3d4a" containerID="a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad" exitCode=0
Sep 08 15:18:40.037124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.037029    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"
Sep 08 15:18:40.037124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.037071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz" event={"ID":"e4c82a4c-a088-4105-a647-eac435db3d4a","Type":"ContainerDied","Data":"a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad"}
Sep 08 15:18:40.037124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.037106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz" event={"ID":"e4c82a4c-a088-4105-a647-eac435db3d4a","Type":"ContainerDied","Data":"9026d826fbcf49d0204965bb940e7e9f4a8da29cf7bb84a794bb2d4213fafde5"}
Sep 08 15:18:40.037124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.037125    2578 scope.go:117] "RemoveContainer" containerID="a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad"
Sep 08 15:18:40.046220 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.046202    2578 scope.go:117] "RemoveContainer" containerID="a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad"
Sep 08 15:18:40.046506 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:18:40.046486    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad\": container with ID starting with a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad not found: ID does not exist" containerID="a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad"
Sep 08 15:18:40.046589 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.046516    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad"} err="failed to get container status \"a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad\": rpc error: code = NotFound desc = could not find container \"a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad\": container with ID starting with a9a99a506588211c0478444e5399af609a876b3e30bb26b826493ccb585accad not found: ID does not exist"
Sep 08 15:18:40.052691 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.052668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"]
Sep 08 15:18:40.054484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.054462    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-ffa941/sub-test-model-ffa941-kserve-67c85b547c-7tgpz"]
Sep 08 15:18:40.301047 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.301003    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd" podUID="c4e52d79-75e1-4fea-b886-20b40734d8a5" containerName="main" containerID="cri-o://6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb" gracePeriod=45
Sep 08 15:18:40.555176 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.555125    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:18:40.650576 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650547    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9nnc\" (UniqueName: \"kubernetes.io/projected/c4e52d79-75e1-4fea-b886-20b40734d8a5-kube-api-access-z9nnc\") pod \"c4e52d79-75e1-4fea-b886-20b40734d8a5\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") "
Sep 08 15:18:40.650718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650586    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-tmp-dir\") pod \"c4e52d79-75e1-4fea-b886-20b40734d8a5\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") "
Sep 08 15:18:40.650718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650632    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-home\") pod \"c4e52d79-75e1-4fea-b886-20b40734d8a5\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") "
Sep 08 15:18:40.650718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650655    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-dshm\") pod \"c4e52d79-75e1-4fea-b886-20b40734d8a5\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") "
Sep 08 15:18:40.650718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650681    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-model-cache\") pod \"c4e52d79-75e1-4fea-b886-20b40734d8a5\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") "
Sep 08 15:18:40.650945 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650726    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e52d79-75e1-4fea-b886-20b40734d8a5-tls-certs\") pod \"c4e52d79-75e1-4fea-b886-20b40734d8a5\" (UID: \"c4e52d79-75e1-4fea-b886-20b40734d8a5\") "
Sep 08 15:18:40.650945 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650845    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c4e52d79-75e1-4fea-b886-20b40734d8a5" (UID: "c4e52d79-75e1-4fea-b886-20b40734d8a5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:40.650945 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650873    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-home" (OuterVolumeSpecName: "home") pod "c4e52d79-75e1-4fea-b886-20b40734d8a5" (UID: "c4e52d79-75e1-4fea-b886-20b40734d8a5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:40.651089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.650972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-model-cache" (OuterVolumeSpecName: "model-cache") pod "c4e52d79-75e1-4fea-b886-20b40734d8a5" (UID: "c4e52d79-75e1-4fea-b886-20b40734d8a5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:40.651089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.651016    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:40.651089 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.651038    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:40.652743 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.652722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e52d79-75e1-4fea-b886-20b40734d8a5-kube-api-access-z9nnc" (OuterVolumeSpecName: "kube-api-access-z9nnc") pod "c4e52d79-75e1-4fea-b886-20b40734d8a5" (UID: "c4e52d79-75e1-4fea-b886-20b40734d8a5"). InnerVolumeSpecName "kube-api-access-z9nnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:18:40.652856 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.652839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-dshm" (OuterVolumeSpecName: "dshm") pod "c4e52d79-75e1-4fea-b886-20b40734d8a5" (UID: "c4e52d79-75e1-4fea-b886-20b40734d8a5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:18:40.652989 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.652970    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4e52d79-75e1-4fea-b886-20b40734d8a5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c4e52d79-75e1-4fea-b886-20b40734d8a5" (UID: "c4e52d79-75e1-4fea-b886-20b40734d8a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:18:40.751785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.751762    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c4e52d79-75e1-4fea-b886-20b40734d8a5-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:40.751785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.751782    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9nnc\" (UniqueName: \"kubernetes.io/projected/c4e52d79-75e1-4fea-b886-20b40734d8a5-kube-api-access-z9nnc\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:40.751905 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.751791    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:40.751905 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:40.751800    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c4e52d79-75e1-4fea-b886-20b40734d8a5-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:18:41.044515 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.044486    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4e52d79-75e1-4fea-b886-20b40734d8a5" containerID="6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb" exitCode=0
Sep 08 15:18:41.044653 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.044546    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"
Sep 08 15:18:41.044653 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.044544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd" event={"ID":"c4e52d79-75e1-4fea-b886-20b40734d8a5","Type":"ContainerDied","Data":"6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb"}
Sep 08 15:18:41.044653 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.044599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd" event={"ID":"c4e52d79-75e1-4fea-b886-20b40734d8a5","Type":"ContainerDied","Data":"23b2f6bbbce488e5878e0d744bd27fa56d7c9c5aae0804d21fd531bc3de73462"}
Sep 08 15:18:41.044653 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.044618    2578 scope.go:117] "RemoveContainer" containerID="6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb"
Sep 08 15:18:41.054508 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.054489    2578 scope.go:117] "RemoveContainer" containerID="6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb"
Sep 08 15:18:41.054879 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:18:41.054859    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb\": container with ID starting with 6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb not found: ID does not exist" containerID="6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb"
Sep 08 15:18:41.054945 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.054889    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb"} err="failed to get container status \"6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb\": rpc error: code = NotFound desc = could not find container \"6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb\": container with ID starting with 6d741108a3ad78c1a71ddf89636d5c53eb5ccef812ad14d5429bb98f7eb90ccb not found: ID does not exist"
Sep 08 15:18:41.065203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.065182    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"]
Sep 08 15:18:41.069702 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.069673    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/sub-test-model-7be222-kserve-cf9b7c7bc-7mrkd"]
Sep 08 15:18:41.833834 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.833802    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4e52d79-75e1-4fea-b886-20b40734d8a5" path="/var/lib/kubelet/pods/c4e52d79-75e1-4fea-b886-20b40734d8a5/volumes"
Sep 08 15:18:41.834204 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:41.834170    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4c82a4c-a088-4105-a647-eac435db3d4a" path="/var/lib/kubelet/pods/e4c82a4c-a088-4105-a647-eac435db3d4a/volumes"
Sep 08 15:18:44.008113 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:44.008085    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:18:55.581564 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:55.581508    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:55.675841 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:55.675813    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:56.637341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:56.637310    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:58.737407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:58.737373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:59.733032 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:59.732999    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:59.885596 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:59.885569    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:18:59.985706 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:18:59.985629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:00.083613 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.083588    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:00.388989 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.388963    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:00.487287 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.487260    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:00.511230 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511203    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"]
Sep 08 15:19:00.511570 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511557    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4e52d79-75e1-4fea-b886-20b40734d8a5" containerName="main"
Sep 08 15:19:00.511622 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511571    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4e52d79-75e1-4fea-b886-20b40734d8a5" containerName="main"
Sep 08 15:19:00.511622 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511597    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4c82a4c-a088-4105-a647-eac435db3d4a" containerName="main"
Sep 08 15:19:00.511622 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511604    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4c82a4c-a088-4105-a647-eac435db3d4a" containerName="main"
Sep 08 15:19:00.511721 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511659    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4e52d79-75e1-4fea-b886-20b40734d8a5" containerName="main"
Sep 08 15:19:00.511721 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.511667    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4c82a4c-a088-4105-a647-eac435db3d4a" containerName="main"
Sep 08 15:19:00.514671 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.514655    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.518896 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.518871    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-0da0d1-dockercfg-6g8vc\""
Sep 08 15:19:00.522821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.519930    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0da0d1\""
Sep 08 15:19:00.529441 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.529419    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"]
Sep 08 15:19:00.559749 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.559727    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"]
Sep 08 15:19:00.562982 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.562967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.573667 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.573647    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"]
Sep 08 15:19:00.605178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.605153    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thl9w\" (UniqueName: \"kubernetes.io/projected/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-kube-api-access-thl9w\") pod \"payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") " pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.605272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.605236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") " pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.706573 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.706493    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") " pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.706573 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.706552    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thl9w\" (UniqueName: \"kubernetes.io/projected/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-kube-api-access-thl9w\") pod \"payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") " pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.706725 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.706602    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brk9b\" (UniqueName: \"kubernetes.io/projected/3f464f9b-b407-4a30-9177-ac5b2ad1045b-kube-api-access-brk9b\") pod \"payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.706725 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.706642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f464f9b-b407-4a30-9177-ac5b2ad1045b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.707157 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.707138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") " pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.715964 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.715940    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thl9w\" (UniqueName: \"kubernetes.io/projected/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-kube-api-access-thl9w\") pod \"payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") " pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.807672 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.807641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brk9b\" (UniqueName: \"kubernetes.io/projected/3f464f9b-b407-4a30-9177-ac5b2ad1045b-kube-api-access-brk9b\") pod \"payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.807805 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.807691    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f464f9b-b407-4a30-9177-ac5b2ad1045b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.808302 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.808284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f464f9b-b407-4a30-9177-ac5b2ad1045b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.815827 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.815804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brk9b\" (UniqueName: \"kubernetes.io/projected/3f464f9b-b407-4a30-9177-ac5b2ad1045b-kube-api-access-brk9b\") pod \"payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:00.828672 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.828653    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:00.872727 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:00.872701    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:01.100586 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:01.100456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"]
Sep 08 15:19:01.102893 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:19:01.102866    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9a8ecf_4ff9_4b8f_a76d_95cbe0549204.slice/crio-42c5544ff94b6dc614407b9f5f3729c06f2a5420218390636c628273daa58d34 WatchSource:0}: Error finding container 42c5544ff94b6dc614407b9f5f3729c06f2a5420218390636c628273daa58d34: Status 404 returned error can't find the container with id 42c5544ff94b6dc614407b9f5f3729c06f2a5420218390636c628273daa58d34
Sep 08 15:19:01.145499 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:01.145476    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf" event={"ID":"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204","Type":"ContainerStarted","Data":"42c5544ff94b6dc614407b9f5f3729c06f2a5420218390636c628273daa58d34"}
Sep 08 15:19:01.157570 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:01.157550    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"]
Sep 08 15:19:01.159456 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:19:01.159432    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f464f9b_b407_4a30_9177_ac5b2ad1045b.slice/crio-0d4823387d5ea6e721e2fb12c4262c4a03a280cf3e0e84a0fe21faf0f1ba010a WatchSource:0}: Error finding container 0d4823387d5ea6e721e2fb12c4262c4a03a280cf3e0e84a0fe21faf0f1ba010a: Status 404 returned error can't find the container with id 0d4823387d5ea6e721e2fb12c4262c4a03a280cf3e0e84a0fe21faf0f1ba010a
Sep 08 15:19:02.153001 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.152965    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq" event={"ID":"3f464f9b-b407-4a30-9177-ac5b2ad1045b","Type":"ContainerStarted","Data":"7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224"}
Sep 08 15:19:02.153439 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.153023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq" event={"ID":"3f464f9b-b407-4a30-9177-ac5b2ad1045b","Type":"ContainerStarted","Data":"0d4823387d5ea6e721e2fb12c4262c4a03a280cf3e0e84a0fe21faf0f1ba010a"}
Sep 08 15:19:02.153439 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.153056    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:02.154774 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.154745    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf" event={"ID":"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204","Type":"ContainerStarted","Data":"b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2"}
Sep 08 15:19:02.154908 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.154840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:02.169363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.169316    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq" podStartSLOduration=2.169299235 podStartE2EDuration="2.169299235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:02.16826595 +0000 UTC m=+1420.819690258" watchObservedRunningTime="2026-09-08 15:19:02.169299235 +0000 UTC m=+1420.820723547"
Sep 08 15:19:02.183379 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:02.183339    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf" podStartSLOduration=2.183327315 podStartE2EDuration="2.183327315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:02.182682442 +0000 UTC m=+1420.834106751" watchObservedRunningTime="2026-09-08 15:19:02.183327315 +0000 UTC m=+1420.834751624"
Sep 08 15:19:05.921236 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:05.921202    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:06.044978 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:06.044953    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:06.146662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:06.146636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:13.162641 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:13.162602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:19:13.163092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:13.162683    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:19:20.882929 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:20.882902    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:23.831740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:23.831703    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:23.931667 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:23.931636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:24.414207 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.414178    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"]
Sep 08 15:19:24.419228 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.419202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.422657 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.422633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"default-dockercfg-zm699\""
Sep 08 15:19:24.422792 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.422671    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"kube-root-ca.crt\""
Sep 08 15:19:24.422792 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.422690    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"openshift-service-ca.crt\""
Sep 08 15:19:24.422792 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.422633    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7be222\"/\"rate-test-model-7be222-kserve-self-signed-certs\""
Sep 08 15:19:24.428990 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.428967    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"]
Sep 08 15:19:24.455982 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.455959    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:24.593319 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.593288    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-tmp-dir\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.593484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.593334    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-dshm\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.593484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.593354    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc9b3df-f079-4b13-acea-030c5d767aa7-tls-certs\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.593484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.593379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glkgv\" (UniqueName: \"kubernetes.io/projected/9dc9b3df-f079-4b13-acea-030c5d767aa7-kube-api-access-glkgv\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.593484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.593409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-home\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.593484 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.593426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-model-cache\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694353 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694284    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-home\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694353 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694312    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-model-cache\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-tmp-dir\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694415    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-dshm\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc9b3df-f079-4b13-acea-030c5d767aa7-tls-certs\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glkgv\" (UniqueName: \"kubernetes.io/projected/9dc9b3df-f079-4b13-acea-030c5d767aa7-kube-api-access-glkgv\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694766 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-home\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-model-cache\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.694825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.694797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-tmp-dir\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.696936 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.696918    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-dshm\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.697098 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.697081    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc9b3df-f079-4b13-acea-030c5d767aa7-tls-certs\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.702067 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.702048    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glkgv\" (UniqueName: \"kubernetes.io/projected/9dc9b3df-f079-4b13-acea-030c5d767aa7-kube-api-access-glkgv\") pod \"rate-test-model-7be222-kserve-784f4996c-kwltz\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") " pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:24.730143 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:24.730118    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:25.000294 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:25.000161    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"]
Sep 08 15:19:25.002892 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:19:25.002868    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dc9b3df_f079_4b13_acea_030c5d767aa7.slice/crio-f67264dcd63c7672a5236d5ad49a3b1ef891203d90f190f99e5166e8e1144a67 WatchSource:0}: Error finding container f67264dcd63c7672a5236d5ad49a3b1ef891203d90f190f99e5166e8e1144a67: Status 404 returned error can't find the container with id f67264dcd63c7672a5236d5ad49a3b1ef891203d90f190f99e5166e8e1144a67
Sep 08 15:19:25.269627 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:25.269599    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz" event={"ID":"9dc9b3df-f079-4b13-acea-030c5d767aa7","Type":"ContainerStarted","Data":"a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6"}
Sep 08 15:19:25.269761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:25.269640    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz" event={"ID":"9dc9b3df-f079-4b13-acea-030c5d767aa7","Type":"ContainerStarted","Data":"f67264dcd63c7672a5236d5ad49a3b1ef891203d90f190f99e5166e8e1144a67"}
Sep 08 15:19:25.287324 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:25.287276    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz" podStartSLOduration=1.2872604779999999 podStartE2EDuration="1.287260478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:19:25.285207029 +0000 UTC m=+1443.936631338" watchObservedRunningTime="2026-09-08 15:19:25.287260478 +0000 UTC m=+1443.938684786"
Sep 08 15:19:26.666462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:26.666429    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:27.378934 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:27.378901    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:34.618941 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:34.618908    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:34.730800 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:34.730772    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:34.730961 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:34.730816    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:34.743481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:34.743457    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:35.328565 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:35.328512    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:19:36.681450 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:36.681418    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:42.416170 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.416137    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:42.446125 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.446102    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"]
Sep 08 15:19:42.446345 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.446306    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq" podUID="c35fdc23-d569-4583-acb4-5f0bd8e80173" containerName="payload-processing" containerID="cri-o://dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7" gracePeriod=30
Sep 08 15:19:42.485345 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.485321    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:42.584892 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.584854    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:42.653945 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.653903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:42.702778 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.702759    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:19:42.742021 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.742000    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fdc23-d569-4583-acb4-5f0bd8e80173-plugins-config-volume\") pod \"c35fdc23-d569-4583-acb4-5f0bd8e80173\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") "
Sep 08 15:19:42.742131 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.742028    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67h25\" (UniqueName: \"kubernetes.io/projected/c35fdc23-d569-4583-acb4-5f0bd8e80173-kube-api-access-67h25\") pod \"c35fdc23-d569-4583-acb4-5f0bd8e80173\" (UID: \"c35fdc23-d569-4583-acb4-5f0bd8e80173\") "
Sep 08 15:19:42.742343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.742322    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c35fdc23-d569-4583-acb4-5f0bd8e80173-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c35fdc23-d569-4583-acb4-5f0bd8e80173" (UID: "c35fdc23-d569-4583-acb4-5f0bd8e80173"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:19:42.744369 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.744343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c35fdc23-d569-4583-acb4-5f0bd8e80173-kube-api-access-67h25" (OuterVolumeSpecName: "kube-api-access-67h25") pod "c35fdc23-d569-4583-acb4-5f0bd8e80173" (UID: "c35fdc23-d569-4583-acb4-5f0bd8e80173"). InnerVolumeSpecName "kube-api-access-67h25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:19:42.843155 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.843132    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67h25\" (UniqueName: \"kubernetes.io/projected/c35fdc23-d569-4583-acb4-5f0bd8e80173-kube-api-access-67h25\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:19:42.843155 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:42.843153    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c35fdc23-d569-4583-acb4-5f0bd8e80173-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:19:43.005458 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.005396    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:43.356343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.356313    2578 generic.go:356] "Generic (PLEG): container finished" podID="c35fdc23-d569-4583-acb4-5f0bd8e80173" containerID="dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7" exitCode=0
Sep 08 15:19:43.356462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.356378    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"
Sep 08 15:19:43.356462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.356404    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq" event={"ID":"c35fdc23-d569-4583-acb4-5f0bd8e80173","Type":"ContainerDied","Data":"dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7"}
Sep 08 15:19:43.356586 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.356463    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq" event={"ID":"c35fdc23-d569-4583-acb4-5f0bd8e80173","Type":"ContainerDied","Data":"587762e320a1f52b52a59103041371feaee06d90035025a162167098c0bc28b2"}
Sep 08 15:19:43.356586 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.356490    2578 scope.go:117] "RemoveContainer" containerID="dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7"
Sep 08 15:19:43.368648 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.368626    2578 scope.go:117] "RemoveContainer" containerID="dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7"
Sep 08 15:19:43.368944 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:19:43.368924    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7\": container with ID starting with dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7 not found: ID does not exist" containerID="dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7"
Sep 08 15:19:43.368997 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.368952    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7"} err="failed to get container status \"dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7\": rpc error: code = NotFound desc = could not find container \"dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7\": container with ID starting with dd16ddcc62e3101898119819f9a952d1faf99677618855df1f179449dfb815a7 not found: ID does not exist"
Sep 08 15:19:43.385474 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.385448    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"]
Sep 08 15:19:43.389799 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.389778    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3ccde3-787b5886d5-5tdjq"]
Sep 08 15:19:43.833537 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:43.833498    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c35fdc23-d569-4583-acb4-5f0bd8e80173" path="/var/lib/kubelet/pods/c35fdc23-d569-4583-acb4-5f0bd8e80173/volumes"
Sep 08 15:19:49.346469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:49.346438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:49.448570 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:49.448515    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:49.655631 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:49.655546    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:51.690385 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:51.690351    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:19:51.791434 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:19:51.791402    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:00.234427 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:00.234392    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:05.888833 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:05.888790    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:05.932082 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:05.932039    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"]
Sep 08 15:20:05.932333 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:05.932298    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf" podUID="ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" containerName="payload-processing" containerID="cri-o://b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2" gracePeriod=30
Sep 08 15:20:05.940462 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:05.940439    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"]
Sep 08 15:20:05.940677 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:05.940656    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq" podUID="3f464f9b-b407-4a30-9177-ac5b2ad1045b" containerName="payload-pre-processing" containerID="cri-o://7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224" gracePeriod=30
Sep 08 15:20:05.979474 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:05.979450    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:06.070536 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.070485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:06.189173 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.189143    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:06.199020 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.199000    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:20:06.215092 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.215073    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:20:06.336010 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.335983    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brk9b\" (UniqueName: \"kubernetes.io/projected/3f464f9b-b407-4a30-9177-ac5b2ad1045b-kube-api-access-brk9b\") pod \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") "
Sep 08 15:20:06.336191 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.336038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thl9w\" (UniqueName: \"kubernetes.io/projected/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-kube-api-access-thl9w\") pod \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") "
Sep 08 15:20:06.336191 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.336057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f464f9b-b407-4a30-9177-ac5b2ad1045b-plugins-config-volume\") pod \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\" (UID: \"3f464f9b-b407-4a30-9177-ac5b2ad1045b\") "
Sep 08 15:20:06.336191 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.336089    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-plugins-config-volume\") pod \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\" (UID: \"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204\") "
Sep 08 15:20:06.336465 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.336443    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f464f9b-b407-4a30-9177-ac5b2ad1045b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f464f9b-b407-4a30-9177-ac5b2ad1045b" (UID: "3f464f9b-b407-4a30-9177-ac5b2ad1045b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:06.336569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.336467    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" (UID: "ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:06.338378 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.338350    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-kube-api-access-thl9w" (OuterVolumeSpecName: "kube-api-access-thl9w") pod "ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" (UID: "ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204"). InnerVolumeSpecName "kube-api-access-thl9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:06.338490 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.338382    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f464f9b-b407-4a30-9177-ac5b2ad1045b-kube-api-access-brk9b" (OuterVolumeSpecName: "kube-api-access-brk9b") pod "3f464f9b-b407-4a30-9177-ac5b2ad1045b" (UID: "3f464f9b-b407-4a30-9177-ac5b2ad1045b"). InnerVolumeSpecName "kube-api-access-brk9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:06.424434 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.424335    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:06.437534 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.437498    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brk9b\" (UniqueName: \"kubernetes.io/projected/3f464f9b-b407-4a30-9177-ac5b2ad1045b-kube-api-access-brk9b\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:06.437602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.437518    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thl9w\" (UniqueName: \"kubernetes.io/projected/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-kube-api-access-thl9w\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:06.437602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.437551    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f464f9b-b407-4a30-9177-ac5b2ad1045b-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:06.437602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.437560    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:06.476177 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.476145    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f464f9b-b407-4a30-9177-ac5b2ad1045b" containerID="7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224" exitCode=0
Sep 08 15:20:06.476284 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.476190    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"
Sep 08 15:20:06.476284 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.476227    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq" event={"ID":"3f464f9b-b407-4a30-9177-ac5b2ad1045b","Type":"ContainerDied","Data":"7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224"}
Sep 08 15:20:06.476284 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.476277    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq" event={"ID":"3f464f9b-b407-4a30-9177-ac5b2ad1045b","Type":"ContainerDied","Data":"0d4823387d5ea6e721e2fb12c4262c4a03a280cf3e0e84a0fe21faf0f1ba010a"}
Sep 08 15:20:06.476474 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.476307    2578 scope.go:117] "RemoveContainer" containerID="7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224"
Sep 08 15:20:06.477628 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.477606    2578 generic.go:356] "Generic (PLEG): container finished" podID="ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" containerID="b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2" exitCode=0
Sep 08 15:20:06.477718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.477681    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf" event={"ID":"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204","Type":"ContainerDied","Data":"b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2"}
Sep 08 15:20:06.477954 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.477732    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf" event={"ID":"ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204","Type":"ContainerDied","Data":"42c5544ff94b6dc614407b9f5f3729c06f2a5420218390636c628273daa58d34"}
Sep 08 15:20:06.477954 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.477740    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"
Sep 08 15:20:06.484882 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.484864    2578 scope.go:117] "RemoveContainer" containerID="7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224"
Sep 08 15:20:06.485182 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:20:06.485161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224\": container with ID starting with 7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224 not found: ID does not exist" containerID="7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224"
Sep 08 15:20:06.485285 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.485190    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224"} err="failed to get container status \"7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224\": rpc error: code = NotFound desc = could not find container \"7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224\": container with ID starting with 7ba6f9c439a312623337fb0d213b72bc0adb25591e0eb8f3c55dabe66d02b224 not found: ID does not exist"
Sep 08 15:20:06.485285 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.485215    2578 scope.go:117] "RemoveContainer" containerID="b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2"
Sep 08 15:20:06.493433 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.493418    2578 scope.go:117] "RemoveContainer" containerID="b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2"
Sep 08 15:20:06.493688 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:20:06.493666    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2\": container with ID starting with b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2 not found: ID does not exist" containerID="b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2"
Sep 08 15:20:06.493749 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.493696    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2"} err="failed to get container status \"b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2\": rpc error: code = NotFound desc = could not find container \"b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2\": container with ID starting with b566b92d13855ff0ff3837309e26156e4cfd1e0878dd793acca5b580256ab4b2 not found: ID does not exist"
Sep 08 15:20:06.529220 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.529197    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"]
Sep 08 15:20:06.543761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.543729    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0da0d1-5d96f4955d-kwfmq"]
Sep 08 15:20:06.567666 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.567637    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"]
Sep 08 15:20:06.573754 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.573736    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0da0d1-55ff7c977c-h95lf"]
Sep 08 15:20:06.587125 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:06.587103    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:07.831969 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:07.831934    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f464f9b-b407-4a30-9177-ac5b2ad1045b" path="/var/lib/kubelet/pods/3f464f9b-b407-4a30-9177-ac5b2ad1045b/volumes"
Sep 08 15:20:07.832339 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:07.832263    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" path="/var/lib/kubelet/pods/ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204/volumes"
Sep 08 15:20:13.835423 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:13.835386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:14.915302 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:14.915253    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:23.304832 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:23.304791    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:23.851004 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:23.850972    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:23.921424 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:23.921386    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:24.020873 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:24.020840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:24.093818 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:24.093792    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:24.408575 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:24.408549    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:25.266946 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:25.266912    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"]
Sep 08 15:20:25.284995 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:25.284967    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:25.708631 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:25.708556    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:26.023366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:26.023326    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:26.215663 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:26.215636    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:29.254228 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:29.254203    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:29.341772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:29.341738    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:30.411344 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:30.411313    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:30.511332 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:30.511302    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:30.658662 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:30.658635    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:30.760517 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:30.760485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:30.904763 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:30.904727    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:30.975285 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:30.975256    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:31.055451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.055380    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"]
Sep 08 15:20:31.055979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.055959    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" containerName="payload-processing"
Sep 08 15:20:31.056023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.055985    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" containerName="payload-processing"
Sep 08 15:20:31.056023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.055998    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f464f9b-b407-4a30-9177-ac5b2ad1045b" containerName="payload-pre-processing"
Sep 08 15:20:31.056023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.056007    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f464f9b-b407-4a30-9177-ac5b2ad1045b" containerName="payload-pre-processing"
Sep 08 15:20:31.056117 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.056050    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35fdc23-d569-4583-acb4-5f0bd8e80173" containerName="payload-processing"
Sep 08 15:20:31.056117 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.056059    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35fdc23-d569-4583-acb4-5f0bd8e80173" containerName="payload-processing"
Sep 08 15:20:31.056184 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.056164    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec9a8ecf-4ff9-4b8f-a76d-95cbe0549204" containerName="payload-processing"
Sep 08 15:20:31.056184 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.056178    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f464f9b-b407-4a30-9177-ac5b2ad1045b" containerName="payload-pre-processing"
Sep 08 15:20:31.056249 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.056189    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35fdc23-d569-4583-acb4-5f0bd8e80173" containerName="payload-processing"
Sep 08 15:20:31.061026 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.061003    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.063400 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.063372    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bdd5c7\""
Sep 08 15:20:31.063501 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.063384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bdd5c7-dockercfg-d4rk9\""
Sep 08 15:20:31.072140 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.072119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"]
Sep 08 15:20:31.236838 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.236795    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brcsn\" (UniqueName: \"kubernetes.io/projected/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-kube-api-access-brcsn\") pod \"payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.236974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.236846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.338420 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.338349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brcsn\" (UniqueName: \"kubernetes.io/projected/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-kube-api-access-brcsn\") pod \"payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.338420 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.338387    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.339049 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.339032    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.346411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.346383    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brcsn\" (UniqueName: \"kubernetes.io/projected/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-kube-api-access-brcsn\") pod \"payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.371811 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.371783    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:31.662104 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:31.662075    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"]
Sep 08 15:20:31.663682 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:20:31.663659    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d5b17f_dbae_4927_9c7e_443fa6f70cf5.slice/crio-b74fa813fd7d415474dbc971c1ac3a6cad3d2a19fa89ff7078e1084e12ba3130 WatchSource:0}: Error finding container b74fa813fd7d415474dbc971c1ac3a6cad3d2a19fa89ff7078e1084e12ba3130: Status 404 returned error can't find the container with id b74fa813fd7d415474dbc971c1ac3a6cad3d2a19fa89ff7078e1084e12ba3130
Sep 08 15:20:32.613291 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:32.613258    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7" event={"ID":"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5","Type":"ContainerStarted","Data":"a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40"}
Sep 08 15:20:32.613467 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:32.613307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7" event={"ID":"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5","Type":"ContainerStarted","Data":"b74fa813fd7d415474dbc971c1ac3a6cad3d2a19fa89ff7078e1084e12ba3130"}
Sep 08 15:20:32.613467 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:32.613387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:32.632149 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:32.632106    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7" podStartSLOduration=1.63209385 podStartE2EDuration="1.63209385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:20:32.631043789 +0000 UTC m=+1511.282468097" watchObservedRunningTime="2026-09-08 15:20:32.63209385 +0000 UTC m=+1511.283518216"
Sep 08 15:20:32.921604 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:32.921537    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:33.024616 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:33.024584    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:40.288979 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.288932    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz" podUID="9dc9b3df-f079-4b13-acea-030c5d767aa7" containerName="main" containerID="cri-o://a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6" gracePeriod=45
Sep 08 15:20:40.545605 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.545550    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:20:40.615680 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615654    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-home\") pod \"9dc9b3df-f079-4b13-acea-030c5d767aa7\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") "
Sep 08 15:20:40.615790 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-dshm\") pod \"9dc9b3df-f079-4b13-acea-030c5d767aa7\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") "
Sep 08 15:20:40.615790 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615756    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-tmp-dir\") pod \"9dc9b3df-f079-4b13-acea-030c5d767aa7\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") "
Sep 08 15:20:40.615790 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615773    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc9b3df-f079-4b13-acea-030c5d767aa7-tls-certs\") pod \"9dc9b3df-f079-4b13-acea-030c5d767aa7\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") "
Sep 08 15:20:40.615965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615792    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glkgv\" (UniqueName: \"kubernetes.io/projected/9dc9b3df-f079-4b13-acea-030c5d767aa7-kube-api-access-glkgv\") pod \"9dc9b3df-f079-4b13-acea-030c5d767aa7\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") "
Sep 08 15:20:40.615965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615807    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-model-cache\") pod \"9dc9b3df-f079-4b13-acea-030c5d767aa7\" (UID: \"9dc9b3df-f079-4b13-acea-030c5d767aa7\") "
Sep 08 15:20:40.615965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.615897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-home" (OuterVolumeSpecName: "home") pod "9dc9b3df-f079-4b13-acea-030c5d767aa7" (UID: "9dc9b3df-f079-4b13-acea-030c5d767aa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:20:40.616117 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.616020    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9dc9b3df-f079-4b13-acea-030c5d767aa7" (UID: "9dc9b3df-f079-4b13-acea-030c5d767aa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:20:40.616117 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.616092    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-model-cache" (OuterVolumeSpecName: "model-cache") pod "9dc9b3df-f079-4b13-acea-030c5d767aa7" (UID: "9dc9b3df-f079-4b13-acea-030c5d767aa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:20:40.616203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.616129    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:40.616203 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.616146    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:40.618006 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.617978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dc9b3df-f079-4b13-acea-030c5d767aa7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9dc9b3df-f079-4b13-acea-030c5d767aa7" (UID: "9dc9b3df-f079-4b13-acea-030c5d767aa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:20:40.618006 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.617993    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-dshm" (OuterVolumeSpecName: "dshm") pod "9dc9b3df-f079-4b13-acea-030c5d767aa7" (UID: "9dc9b3df-f079-4b13-acea-030c5d767aa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:20:40.618145 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.618030    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc9b3df-f079-4b13-acea-030c5d767aa7-kube-api-access-glkgv" (OuterVolumeSpecName: "kube-api-access-glkgv") pod "9dc9b3df-f079-4b13-acea-030c5d767aa7" (UID: "9dc9b3df-f079-4b13-acea-030c5d767aa7"). InnerVolumeSpecName "kube-api-access-glkgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:40.656825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.656786    2578 generic.go:356] "Generic (PLEG): container finished" podID="9dc9b3df-f079-4b13-acea-030c5d767aa7" containerID="a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6" exitCode=0
Sep 08 15:20:40.656928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.656831    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"
Sep 08 15:20:40.656928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.656874    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz" event={"ID":"9dc9b3df-f079-4b13-acea-030c5d767aa7","Type":"ContainerDied","Data":"a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6"}
Sep 08 15:20:40.656928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.656911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz" event={"ID":"9dc9b3df-f079-4b13-acea-030c5d767aa7","Type":"ContainerDied","Data":"f67264dcd63c7672a5236d5ad49a3b1ef891203d90f190f99e5166e8e1144a67"}
Sep 08 15:20:40.657061 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.656933    2578 scope.go:117] "RemoveContainer" containerID="a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6"
Sep 08 15:20:40.665999 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.665982    2578 scope.go:117] "RemoveContainer" containerID="a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6"
Sep 08 15:20:40.666254 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:20:40.666236    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6\": container with ID starting with a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6 not found: ID does not exist" containerID="a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6"
Sep 08 15:20:40.666319 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.666261    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6"} err="failed to get container status \"a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6\": rpc error: code = NotFound desc = could not find container \"a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6\": container with ID starting with a03f0d36f667c9a6bf13315e775080940a348fb07c654fb9b476b9eb7c92ada6 not found: ID does not exist"
Sep 08 15:20:40.678908 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.678884    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"]
Sep 08 15:20:40.684098 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.684080    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7be222/rate-test-model-7be222-kserve-784f4996c-kwltz"]
Sep 08 15:20:40.716842 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.716826    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:40.716842 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.716842    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9dc9b3df-f079-4b13-acea-030c5d767aa7-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:40.716947 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.716851    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glkgv\" (UniqueName: \"kubernetes.io/projected/9dc9b3df-f079-4b13-acea-030c5d767aa7-kube-api-access-glkgv\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:40.716947 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:40.716860    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9dc9b3df-f079-4b13-acea-030c5d767aa7-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:41.560422 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:41.560389    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:41.833317 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:41.833242    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dc9b3df-f079-4b13-acea-030c5d767aa7" path="/var/lib/kubelet/pods/9dc9b3df-f079-4b13-acea-030c5d767aa7/volumes"
Sep 08 15:20:42.193675 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.193598    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:42.214458 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.214420    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"]
Sep 08 15:20:42.214686 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.214661    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb" podUID="b12412a1-3a49-43f1-8680-8fd68bee245c" containerName="payload-processing" containerID="cri-o://bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9" gracePeriod=30
Sep 08 15:20:42.222624 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.222594    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"]
Sep 08 15:20:42.222880 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.222850    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q" podUID="d8acf928-2d49-4120-aa6b-6147d156b7d8" containerName="payload-pre-processing" containerID="cri-o://224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9" gracePeriod=30
Sep 08 15:20:42.269477 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.269454    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:42.369024 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.368980    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:42.480609 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.480588    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:20:42.497681 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.497663    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:20:42.534233 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.534211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12412a1-3a49-43f1-8680-8fd68bee245c-plugins-config-volume\") pod \"b12412a1-3a49-43f1-8680-8fd68bee245c\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") "
Sep 08 15:20:42.534349 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.534296    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9z97\" (UniqueName: \"kubernetes.io/projected/b12412a1-3a49-43f1-8680-8fd68bee245c-kube-api-access-z9z97\") pod \"b12412a1-3a49-43f1-8680-8fd68bee245c\" (UID: \"b12412a1-3a49-43f1-8680-8fd68bee245c\") "
Sep 08 15:20:42.534349 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.534344    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sjwr\" (UniqueName: \"kubernetes.io/projected/d8acf928-2d49-4120-aa6b-6147d156b7d8-kube-api-access-2sjwr\") pod \"d8acf928-2d49-4120-aa6b-6147d156b7d8\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") "
Sep 08 15:20:42.534461 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.534392    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8acf928-2d49-4120-aa6b-6147d156b7d8-plugins-config-volume\") pod \"d8acf928-2d49-4120-aa6b-6147d156b7d8\" (UID: \"d8acf928-2d49-4120-aa6b-6147d156b7d8\") "
Sep 08 15:20:42.534750 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.534723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12412a1-3a49-43f1-8680-8fd68bee245c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b12412a1-3a49-43f1-8680-8fd68bee245c" (UID: "b12412a1-3a49-43f1-8680-8fd68bee245c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:42.534868 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.534765    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8acf928-2d49-4120-aa6b-6147d156b7d8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d8acf928-2d49-4120-aa6b-6147d156b7d8" (UID: "d8acf928-2d49-4120-aa6b-6147d156b7d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:20:42.536422 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.536400    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12412a1-3a49-43f1-8680-8fd68bee245c-kube-api-access-z9z97" (OuterVolumeSpecName: "kube-api-access-z9z97") pod "b12412a1-3a49-43f1-8680-8fd68bee245c" (UID: "b12412a1-3a49-43f1-8680-8fd68bee245c"). InnerVolumeSpecName "kube-api-access-z9z97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:42.536549 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.536469    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8acf928-2d49-4120-aa6b-6147d156b7d8-kube-api-access-2sjwr" (OuterVolumeSpecName: "kube-api-access-2sjwr") pod "d8acf928-2d49-4120-aa6b-6147d156b7d8" (UID: "d8acf928-2d49-4120-aa6b-6147d156b7d8"). InnerVolumeSpecName "kube-api-access-2sjwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:20:42.634996 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.634976    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9z97\" (UniqueName: \"kubernetes.io/projected/b12412a1-3a49-43f1-8680-8fd68bee245c-kube-api-access-z9z97\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:42.634996 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.634996    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sjwr\" (UniqueName: \"kubernetes.io/projected/d8acf928-2d49-4120-aa6b-6147d156b7d8-kube-api-access-2sjwr\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:42.635341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.635005    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8acf928-2d49-4120-aa6b-6147d156b7d8-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:42.635341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.635014    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b12412a1-3a49-43f1-8680-8fd68bee245c-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:20:42.669451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.669422    2578 generic.go:356] "Generic (PLEG): container finished" podID="d8acf928-2d49-4120-aa6b-6147d156b7d8" containerID="224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9" exitCode=0
Sep 08 15:20:42.669564 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.669462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"
Sep 08 15:20:42.669564 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.669513    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q" event={"ID":"d8acf928-2d49-4120-aa6b-6147d156b7d8","Type":"ContainerDied","Data":"224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9"}
Sep 08 15:20:42.669684 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.669608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q" event={"ID":"d8acf928-2d49-4120-aa6b-6147d156b7d8","Type":"ContainerDied","Data":"0b039c81af21f67e44f0ea2408c222d6e044a00669ff12a84127c6c9cca28bf6"}
Sep 08 15:20:42.669684 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.669634    2578 scope.go:117] "RemoveContainer" containerID="224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9"
Sep 08 15:20:42.671132 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.671108    2578 generic.go:356] "Generic (PLEG): container finished" podID="b12412a1-3a49-43f1-8680-8fd68bee245c" containerID="bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9" exitCode=0
Sep 08 15:20:42.671234 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.671148    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"
Sep 08 15:20:42.671234 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.671187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb" event={"ID":"b12412a1-3a49-43f1-8680-8fd68bee245c","Type":"ContainerDied","Data":"bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9"}
Sep 08 15:20:42.671329 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.671249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb" event={"ID":"b12412a1-3a49-43f1-8680-8fd68bee245c","Type":"ContainerDied","Data":"b05c9c301b4a953829371eb588f1dba74a637d36217f3643ff60fa5e58c64e0d"}
Sep 08 15:20:42.678374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.678360    2578 scope.go:117] "RemoveContainer" containerID="224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9"
Sep 08 15:20:42.678669 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:20:42.678641    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9\": container with ID starting with 224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9 not found: ID does not exist" containerID="224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9"
Sep 08 15:20:42.678772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.678675    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9"} err="failed to get container status \"224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9\": rpc error: code = NotFound desc = could not find container \"224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9\": container with ID starting with 224455bf36754cb64c6c0139abe9b329a8af1c7b29addd9928995ea6b2bdb3e9 not found: ID does not exist"
Sep 08 15:20:42.678772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.678694    2578 scope.go:117] "RemoveContainer" containerID="bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9"
Sep 08 15:20:42.686836 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.686821    2578 scope.go:117] "RemoveContainer" containerID="bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9"
Sep 08 15:20:42.687100 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:20:42.687081    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9\": container with ID starting with bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9 not found: ID does not exist" containerID="bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9"
Sep 08 15:20:42.687171 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.687107    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9"} err="failed to get container status \"bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9\": rpc error: code = NotFound desc = could not find container \"bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9\": container with ID starting with bb81a1055b7a08c4914b11d66c6cb095790d08e91f5f3953952263498ea691f9 not found: ID does not exist"
Sep 08 15:20:42.699020 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.699001    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"]
Sep 08 15:20:42.702893 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.702875    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-65dd51-7c8d964866-6zr2q"]
Sep 08 15:20:42.714076 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.714053    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"]
Sep 08 15:20:42.719353 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:42.719306    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-65dd51-66f7b99477-s2prb"]
Sep 08 15:20:43.208933 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:43.208904    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:43.262774 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:43.262723    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:43.620195 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:43.620176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:20:43.832081 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:43.832049    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12412a1-3a49-43f1-8680-8fd68bee245c" path="/var/lib/kubelet/pods/b12412a1-3a49-43f1-8680-8fd68bee245c/volumes"
Sep 08 15:20:43.832450 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:43.832369    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8acf928-2d49-4120-aa6b-6147d156b7d8" path="/var/lib/kubelet/pods/d8acf928-2d49-4120-aa6b-6147d156b7d8/volumes"
Sep 08 15:20:48.693654 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:48.693623    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:51.079027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:51.078994    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:51.431079 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:51.431001    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:54.237987 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:54.237941    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:54.911006 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:54.910976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:55.232405 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:55.232303    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:55.327305 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:55.327249    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:58.420877 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:58.420847    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:58.511622 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:58.511592    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:58.827781 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:58.827753    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:59.635267 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:59.635220    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:20:59.737469 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:20:59.737442    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:00.999881 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:00.999845    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:01.100660 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:01.100629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:03.333150 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:03.333113    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:04.274543 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:04.274490    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:23.403222 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:23.403183    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:30.396132 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396088    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"]
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396478    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dc9b3df-f079-4b13-acea-030c5d767aa7" containerName="main"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396491    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dc9b3df-f079-4b13-acea-030c5d767aa7" containerName="main"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396505    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12412a1-3a49-43f1-8680-8fd68bee245c" containerName="payload-processing"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396510    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12412a1-3a49-43f1-8680-8fd68bee245c" containerName="payload-processing"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396559    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8acf928-2d49-4120-aa6b-6147d156b7d8" containerName="payload-pre-processing"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396566    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8acf928-2d49-4120-aa6b-6147d156b7d8" containerName="payload-pre-processing"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396647    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dc9b3df-f079-4b13-acea-030c5d767aa7" containerName="main"
Sep 08 15:21:30.396649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396658    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8acf928-2d49-4120-aa6b-6147d156b7d8" containerName="payload-pre-processing"
Sep 08 15:21:30.396999 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.396665    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12412a1-3a49-43f1-8680-8fd68bee245c" containerName="payload-processing"
Sep 08 15:21:30.400949 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.400928    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.403431 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.403408    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bdd5c7\"/\"openshift-service-ca.crt\""
Sep 08 15:21:30.404213 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.404189    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bdd5c7\"/\"kube-root-ca.crt\""
Sep 08 15:21:30.404325 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.404289    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bdd5c7\"/\"default-dockercfg-jbsvb\""
Sep 08 15:21:30.404403 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.404345    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bdd5c7\"/\"ipp-route-bdd5c7-kserve-self-signed-certs\""
Sep 08 15:21:30.407730 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.407703    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"]
Sep 08 15:21:30.497625 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.497590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-tmp-dir\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.497733 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.497640    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-model-cache\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.497733 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.497697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-home\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.497809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.497750    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-dshm\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.497809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.497775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ssn\" (UniqueName: \"kubernetes.io/projected/b7f0022f-3482-435d-824d-cedb3f6495cb-kube-api-access-75ssn\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.497809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.497805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0022f-3482-435d-824d-cedb3f6495cb-tls-certs\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.598692 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.598671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-dshm\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.598793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.598704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ssn\" (UniqueName: \"kubernetes.io/projected/b7f0022f-3482-435d-824d-cedb3f6495cb-kube-api-access-75ssn\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.598793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.598728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0022f-3482-435d-824d-cedb3f6495cb-tls-certs\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.598793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.598746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-tmp-dir\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.598793 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.598785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-model-cache\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.598997 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.598826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-home\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.599135 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.599108    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-tmp-dir\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.599205 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.599131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-model-cache\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.599205 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.599178    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-home\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.601036 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.601015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-dshm\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.601444 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.601424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0022f-3482-435d-824d-cedb3f6495cb-tls-certs\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.606987 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.606965    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ssn\" (UniqueName: \"kubernetes.io/projected/b7f0022f-3482-435d-824d-cedb3f6495cb-kube-api-access-75ssn\") pod \"ipp-route-bdd5c7-kserve-854dd98855-s6v7w\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") " pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.713331 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.713278    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:30.816608 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.816578    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:30.985329 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:30.985286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"]
Sep 08 15:21:30.987595 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:21:30.987567    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7f0022f_3482_435d_824d_cedb3f6495cb.slice/crio-809f05ff184f4e64aa4d1301d1ca3d1eb59173ed47387c7371fe9d053d838f49 WatchSource:0}: Error finding container 809f05ff184f4e64aa4d1301d1ca3d1eb59173ed47387c7371fe9d053d838f49: Status 404 returned error can't find the container with id 809f05ff184f4e64aa4d1301d1ca3d1eb59173ed47387c7371fe9d053d838f49
Sep 08 15:21:31.907386 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:31.907356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w" event={"ID":"b7f0022f-3482-435d-824d-cedb3f6495cb","Type":"ContainerStarted","Data":"6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85"}
Sep 08 15:21:31.907386 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:31.907393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w" event={"ID":"b7f0022f-3482-435d-824d-cedb3f6495cb","Type":"ContainerStarted","Data":"809f05ff184f4e64aa4d1301d1ca3d1eb59173ed47387c7371fe9d053d838f49"}
Sep 08 15:21:31.924421 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:31.924351    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w" podStartSLOduration=1.924335939 podStartE2EDuration="1.924335939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:21:31.923082234 +0000 UTC m=+1570.574506543" watchObservedRunningTime="2026-09-08 15:21:31.924335939 +0000 UTC m=+1570.575760248"
Sep 08 15:21:32.382242 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:32.382198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:32.437694 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:32.437650    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:21:40.713834 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:40.713792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:40.714508 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:40.713849    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:40.726701 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:40.726673    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:21:40.958632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:21:40.958597    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:22:01.229923 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:01.229891    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7_f5d5b17f-dbae-4927-9c7e-443fa6f70cf5/payload-processing/0.log"
Sep 08 15:22:01.776362 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:01.776329    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:02.105648 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:02.105573    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:02.694771 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:02.694739    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:04.997830 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:04.997799    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:05.098704 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:05.098676    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:06.169041 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.168993    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:06.428183 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.428107    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:06.535714 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.535686    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:06.661171 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.661146    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:06.762655 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.762629    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:06.780381 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.780355    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"]
Sep 08 15:22:06.784119 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.784102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:06.786703 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.786681    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f9a001-dockercfg-rgdhc\""
Sep 08 15:22:06.786820 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.786719    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f9a001\""
Sep 08 15:22:06.797273 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.797243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"]
Sep 08 15:22:06.813498 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.813465    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"]
Sep 08 15:22:06.816503 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.816483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.833105 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.833086    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"]
Sep 08 15:22:06.880599 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.880579    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a172f667-c655-4a50-95ea-8cbc080e0048-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:06.880696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.880609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5614deb9-a1a9-4059-8825-81c69e3de0db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.880743 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.880697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkcqb\" (UniqueName: \"kubernetes.io/projected/a172f667-c655-4a50-95ea-8cbc080e0048-kube-api-access-nkcqb\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:06.880743 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.880726    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqd4l\" (UniqueName: \"kubernetes.io/projected/5614deb9-a1a9-4059-8825-81c69e3de0db-kube-api-access-dqd4l\") pod \"payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.981916 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.981887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkcqb\" (UniqueName: \"kubernetes.io/projected/a172f667-c655-4a50-95ea-8cbc080e0048-kube-api-access-nkcqb\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:06.982027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.981921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqd4l\" (UniqueName: \"kubernetes.io/projected/5614deb9-a1a9-4059-8825-81c69e3de0db-kube-api-access-dqd4l\") pod \"payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.982027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.981977    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a172f667-c655-4a50-95ea-8cbc080e0048-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:06.982027 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.981999    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5614deb9-a1a9-4059-8825-81c69e3de0db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.982712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.982694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a172f667-c655-4a50-95ea-8cbc080e0048-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:06.982879 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.982860    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5614deb9-a1a9-4059-8825-81c69e3de0db-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.989713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.989688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqd4l\" (UniqueName: \"kubernetes.io/projected/5614deb9-a1a9-4059-8825-81c69e3de0db-kube-api-access-dqd4l\") pod \"payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:06.989827 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:06.989723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkcqb\" (UniqueName: \"kubernetes.io/projected/a172f667-c655-4a50-95ea-8cbc080e0048-kube-api-access-nkcqb\") pod \"payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:07.095796 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:07.095726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:07.126644 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:07.126623    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:07.373028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:07.373007    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"]
Sep 08 15:22:07.374862 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:22:07.374833    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda172f667_c655_4a50_95ea_8cbc080e0048.slice/crio-97977744a57ccfd60d0dc9a0de8a211c26e09deaeb6cd4c4bd52e09549c16691 WatchSource:0}: Error finding container 97977744a57ccfd60d0dc9a0de8a211c26e09deaeb6cd4c4bd52e09549c16691: Status 404 returned error can't find the container with id 97977744a57ccfd60d0dc9a0de8a211c26e09deaeb6cd4c4bd52e09549c16691
Sep 08 15:22:07.412766 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:07.412735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"]
Sep 08 15:22:07.413690 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:22:07.413663    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5614deb9_a1a9_4059_8825_81c69e3de0db.slice/crio-9302f2479523b6746ff952416831e0fafe539c746ab9dcee45b321c1b5b2991c WatchSource:0}: Error finding container 9302f2479523b6746ff952416831e0fafe539c746ab9dcee45b321c1b5b2991c: Status 404 returned error can't find the container with id 9302f2479523b6746ff952416831e0fafe539c746ab9dcee45b321c1b5b2991c
Sep 08 15:22:08.078112 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.078073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c" event={"ID":"5614deb9-a1a9-4059-8825-81c69e3de0db","Type":"ContainerStarted","Data":"e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee"}
Sep 08 15:22:08.078293 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.078140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:08.078293 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.078165    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c" event={"ID":"5614deb9-a1a9-4059-8825-81c69e3de0db","Type":"ContainerStarted","Data":"9302f2479523b6746ff952416831e0fafe539c746ab9dcee45b321c1b5b2991c"}
Sep 08 15:22:08.079770 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.079747    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7" event={"ID":"a172f667-c655-4a50-95ea-8cbc080e0048","Type":"ContainerStarted","Data":"09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423"}
Sep 08 15:22:08.079876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.079793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7" event={"ID":"a172f667-c655-4a50-95ea-8cbc080e0048","Type":"ContainerStarted","Data":"97977744a57ccfd60d0dc9a0de8a211c26e09deaeb6cd4c4bd52e09549c16691"}
Sep 08 15:22:08.079876 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.079832    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:08.098291 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.098252    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c" podStartSLOduration=2.098238281 podStartE2EDuration="2.098238281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:22:08.095729064 +0000 UTC m=+1606.747153372" watchObservedRunningTime="2026-09-08 15:22:08.098238281 +0000 UTC m=+1606.749662589"
Sep 08 15:22:08.113997 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:08.113952    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7" podStartSLOduration=2.113938083 podStartE2EDuration="2.113938083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:22:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:22:08.112797669 +0000 UTC m=+1606.764222003" watchObservedRunningTime="2026-09-08 15:22:08.113938083 +0000 UTC m=+1606.765362389"
Sep 08 15:22:19.087710 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:19.087620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:19.088200 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:19.087816    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:29.879575 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:29.879538    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:36.799769 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.799721    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:36.805977 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.805936    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"]
Sep 08 15:22:36.806212 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.806191    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c" podUID="5614deb9-a1a9-4059-8825-81c69e3de0db" containerName="payload-processing" containerID="cri-o://e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee" gracePeriod=30
Sep 08 15:22:36.811726 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.811691    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"]
Sep 08 15:22:36.811961 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.811921    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7" podUID="a172f667-c655-4a50-95ea-8cbc080e0048" containerName="payload-pre-processing" containerID="cri-o://09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423" gracePeriod=30
Sep 08 15:22:36.902728 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.902702    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:36.980390 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:36.980364    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:37.042437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.042406    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:37.072110 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.072091    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:37.089851 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.089819    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:37.223226 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.223194    2578 generic.go:356] "Generic (PLEG): container finished" podID="5614deb9-a1a9-4059-8825-81c69e3de0db" containerID="e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee" exitCode=0
Sep 08 15:22:37.223352 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.223235    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"
Sep 08 15:22:37.223352 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.223286    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c" event={"ID":"5614deb9-a1a9-4059-8825-81c69e3de0db","Type":"ContainerDied","Data":"e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee"}
Sep 08 15:22:37.223352 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.223333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c" event={"ID":"5614deb9-a1a9-4059-8825-81c69e3de0db","Type":"ContainerDied","Data":"9302f2479523b6746ff952416831e0fafe539c746ab9dcee45b321c1b5b2991c"}
Sep 08 15:22:37.223478 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.223360    2578 scope.go:117] "RemoveContainer" containerID="e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee"
Sep 08 15:22:37.224545 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.224501    2578 generic.go:356] "Generic (PLEG): container finished" podID="a172f667-c655-4a50-95ea-8cbc080e0048" containerID="09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423" exitCode=0
Sep 08 15:22:37.224612 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.224566    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"
Sep 08 15:22:37.224649 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.224566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7" event={"ID":"a172f667-c655-4a50-95ea-8cbc080e0048","Type":"ContainerDied","Data":"09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423"}
Sep 08 15:22:37.224680 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.224673    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7" event={"ID":"a172f667-c655-4a50-95ea-8cbc080e0048","Type":"ContainerDied","Data":"97977744a57ccfd60d0dc9a0de8a211c26e09deaeb6cd4c4bd52e09549c16691"}
Sep 08 15:22:37.225735 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.225716    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5614deb9-a1a9-4059-8825-81c69e3de0db-plugins-config-volume\") pod \"5614deb9-a1a9-4059-8825-81c69e3de0db\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") "
Sep 08 15:22:37.225821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.225759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a172f667-c655-4a50-95ea-8cbc080e0048-plugins-config-volume\") pod \"a172f667-c655-4a50-95ea-8cbc080e0048\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") "
Sep 08 15:22:37.225882 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.225830    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkcqb\" (UniqueName: \"kubernetes.io/projected/a172f667-c655-4a50-95ea-8cbc080e0048-kube-api-access-nkcqb\") pod \"a172f667-c655-4a50-95ea-8cbc080e0048\" (UID: \"a172f667-c655-4a50-95ea-8cbc080e0048\") "
Sep 08 15:22:37.225882 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.225874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqd4l\" (UniqueName: \"kubernetes.io/projected/5614deb9-a1a9-4059-8825-81c69e3de0db-kube-api-access-dqd4l\") pod \"5614deb9-a1a9-4059-8825-81c69e3de0db\" (UID: \"5614deb9-a1a9-4059-8825-81c69e3de0db\") "
Sep 08 15:22:37.226163 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.226145    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a172f667-c655-4a50-95ea-8cbc080e0048-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a172f667-c655-4a50-95ea-8cbc080e0048" (UID: "a172f667-c655-4a50-95ea-8cbc080e0048"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:22:37.226163 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.226153    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5614deb9-a1a9-4059-8825-81c69e3de0db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5614deb9-a1a9-4059-8825-81c69e3de0db" (UID: "5614deb9-a1a9-4059-8825-81c69e3de0db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:22:37.228028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.228002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a172f667-c655-4a50-95ea-8cbc080e0048-kube-api-access-nkcqb" (OuterVolumeSpecName: "kube-api-access-nkcqb") pod "a172f667-c655-4a50-95ea-8cbc080e0048" (UID: "a172f667-c655-4a50-95ea-8cbc080e0048"). InnerVolumeSpecName "kube-api-access-nkcqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:22:37.228107 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.228073    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5614deb9-a1a9-4059-8825-81c69e3de0db-kube-api-access-dqd4l" (OuterVolumeSpecName: "kube-api-access-dqd4l") pod "5614deb9-a1a9-4059-8825-81c69e3de0db" (UID: "5614deb9-a1a9-4059-8825-81c69e3de0db"). InnerVolumeSpecName "kube-api-access-dqd4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:22:37.232032 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.232015    2578 scope.go:117] "RemoveContainer" containerID="e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee"
Sep 08 15:22:37.232282 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:22:37.232265    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee\": container with ID starting with e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee not found: ID does not exist" containerID="e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee"
Sep 08 15:22:37.232360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.232293    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee"} err="failed to get container status \"e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee\": rpc error: code = NotFound desc = could not find container \"e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee\": container with ID starting with e79aa89a46fa7b134c4d336ad5310b1e1526af550509839e46f81be143deafee not found: ID does not exist"
Sep 08 15:22:37.232360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.232314    2578 scope.go:117] "RemoveContainer" containerID="09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423"
Sep 08 15:22:37.240028 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.240014    2578 scope.go:117] "RemoveContainer" containerID="09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423"
Sep 08 15:22:37.240272 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:22:37.240254    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423\": container with ID starting with 09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423 not found: ID does not exist" containerID="09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423"
Sep 08 15:22:37.240321 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.240276    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423"} err="failed to get container status \"09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423\": rpc error: code = NotFound desc = could not find container \"09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423\": container with ID starting with 09a75bac02646053cb87c4e57f37152d4aa82c2b9cf65f68885bc75afbe2e423 not found: ID does not exist"
Sep 08 15:22:37.327372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.327323    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkcqb\" (UniqueName: \"kubernetes.io/projected/a172f667-c655-4a50-95ea-8cbc080e0048-kube-api-access-nkcqb\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:22:37.327372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.327344    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqd4l\" (UniqueName: \"kubernetes.io/projected/5614deb9-a1a9-4059-8825-81c69e3de0db-kube-api-access-dqd4l\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:22:37.327372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.327352    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5614deb9-a1a9-4059-8825-81c69e3de0db-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:22:37.327372 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.327360    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a172f667-c655-4a50-95ea-8cbc080e0048-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:22:37.358121 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.358101    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:37.548696 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.548668    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"]
Sep 08 15:22:37.551864 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.551841    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f9a001-5649b6c8-vlqx7"]
Sep 08 15:22:37.565370 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.565343    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"]
Sep 08 15:22:37.571399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.571381    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-f9a001-dd58d49c-4d84c"]
Sep 08 15:22:37.831519 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.831494    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5614deb9-a1a9-4059-8825-81c69e3de0db" path="/var/lib/kubelet/pods/5614deb9-a1a9-4059-8825-81c69e3de0db/volumes"
Sep 08 15:22:37.831941 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:37.831924    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a172f667-c655-4a50-95ea-8cbc080e0048" path="/var/lib/kubelet/pods/a172f667-c655-4a50-95ea-8cbc080e0048/volumes"
Sep 08 15:22:43.463195 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:43.463155    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:43.551058 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:43.551036    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:43.755857 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:43.755776    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:22:47.746466 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:22:47.746438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:03.063540 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.063487    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:03.084928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.084897    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"]
Sep 08 15:23:03.085246 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.085194    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7" podUID="f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" containerName="payload-processing" containerID="cri-o://a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40" gracePeriod=30
Sep 08 15:23:03.129866 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.129840    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:03.231726 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.231689    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:03.293007 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.292977    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:03.337936 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.337917    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:23:03.346330 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.346303    2578 generic.go:356] "Generic (PLEG): container finished" podID="f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" containerID="a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40" exitCode=0
Sep 08 15:23:03.346447 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.346343    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"
Sep 08 15:23:03.346447 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.346388    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7" event={"ID":"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5","Type":"ContainerDied","Data":"a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40"}
Sep 08 15:23:03.346543 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.346451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7" event={"ID":"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5","Type":"ContainerDied","Data":"b74fa813fd7d415474dbc971c1ac3a6cad3d2a19fa89ff7078e1084e12ba3130"}
Sep 08 15:23:03.346543 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.346473    2578 scope.go:117] "RemoveContainer" containerID="a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40"
Sep 08 15:23:03.355861 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.355836    2578 scope.go:117] "RemoveContainer" containerID="a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40"
Sep 08 15:23:03.356198 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:23:03.356161    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40\": container with ID starting with a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40 not found: ID does not exist" containerID="a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40"
Sep 08 15:23:03.356298 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.356199    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40"} err="failed to get container status \"a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40\": rpc error: code = NotFound desc = could not find container \"a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40\": container with ID starting with a55b29fccd3b44b240cd1b7ab42a51247df0fa6440eac980a987371c243b2b40 not found: ID does not exist"
Sep 08 15:23:03.518374 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.518351    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brcsn\" (UniqueName: \"kubernetes.io/projected/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-kube-api-access-brcsn\") pod \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") "
Sep 08 15:23:03.518506 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.518423    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-plugins-config-volume\") pod \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\" (UID: \"f5d5b17f-dbae-4927-9c7e-443fa6f70cf5\") "
Sep 08 15:23:03.518794 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.518764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" (UID: "f5d5b17f-dbae-4927-9c7e-443fa6f70cf5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:23:03.520363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.520341    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-kube-api-access-brcsn" (OuterVolumeSpecName: "kube-api-access-brcsn") pod "f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" (UID: "f5d5b17f-dbae-4927-9c7e-443fa6f70cf5"). InnerVolumeSpecName "kube-api-access-brcsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:23:03.618910 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.618855    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brcsn\" (UniqueName: \"kubernetes.io/projected/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-kube-api-access-brcsn\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:03.618910 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.618879    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:03.628563 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.628539    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:03.668011 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.667987    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"]
Sep 08 15:23:03.672005 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.671987    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bdd5c7-7c447b9f79-5h5l7"]
Sep 08 15:23:03.831123 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:03.831098    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" path="/var/lib/kubelet/pods/f5d5b17f-dbae-4927-9c7e-443fa6f70cf5/volumes"
Sep 08 15:23:08.616549 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:08.615693    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"]
Sep 08 15:23:08.707847 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:08.707817    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:23.638199 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.638139    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w" podUID="b7f0022f-3482-435d-824d-cedb3f6495cb" containerName="main" containerID="cri-o://6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85" gracePeriod=45
Sep 08 15:23:23.884115 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.884092    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:23:23.978955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.978874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-home\") pod \"b7f0022f-3482-435d-824d-cedb3f6495cb\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") "
Sep 08 15:23:23.978955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.978916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0022f-3482-435d-824d-cedb3f6495cb-tls-certs\") pod \"b7f0022f-3482-435d-824d-cedb3f6495cb\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") "
Sep 08 15:23:23.978955 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.978942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-tmp-dir\") pod \"b7f0022f-3482-435d-824d-cedb3f6495cb\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") "
Sep 08 15:23:23.979190 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.978979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-75ssn\" (UniqueName: \"kubernetes.io/projected/b7f0022f-3482-435d-824d-cedb3f6495cb-kube-api-access-75ssn\") pod \"b7f0022f-3482-435d-824d-cedb3f6495cb\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") "
Sep 08 15:23:23.979190 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.979001    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-model-cache\") pod \"b7f0022f-3482-435d-824d-cedb3f6495cb\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") "
Sep 08 15:23:23.979190 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.979015    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-dshm\") pod \"b7f0022f-3482-435d-824d-cedb3f6495cb\" (UID: \"b7f0022f-3482-435d-824d-cedb3f6495cb\") "
Sep 08 15:23:23.979360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.979187    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-home" (OuterVolumeSpecName: "home") pod "b7f0022f-3482-435d-824d-cedb3f6495cb" (UID: "b7f0022f-3482-435d-824d-cedb3f6495cb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:23:23.979360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.979245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7f0022f-3482-435d-824d-cedb3f6495cb" (UID: "b7f0022f-3482-435d-824d-cedb3f6495cb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:23:23.979360 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.979274    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7f0022f-3482-435d-824d-cedb3f6495cb" (UID: "b7f0022f-3482-435d-824d-cedb3f6495cb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:23:23.981244 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.981218    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-dshm" (OuterVolumeSpecName: "dshm") pod "b7f0022f-3482-435d-824d-cedb3f6495cb" (UID: "b7f0022f-3482-435d-824d-cedb3f6495cb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:23:23.981341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.981263    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f0022f-3482-435d-824d-cedb3f6495cb-kube-api-access-75ssn" (OuterVolumeSpecName: "kube-api-access-75ssn") pod "b7f0022f-3482-435d-824d-cedb3f6495cb" (UID: "b7f0022f-3482-435d-824d-cedb3f6495cb"). InnerVolumeSpecName "kube-api-access-75ssn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:23:23.981341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:23.981302    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f0022f-3482-435d-824d-cedb3f6495cb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7f0022f-3482-435d-824d-cedb3f6495cb" (UID: "b7f0022f-3482-435d-824d-cedb3f6495cb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:23:24.080449 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.080420    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:24.080449 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.080443    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f0022f-3482-435d-824d-cedb3f6495cb-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:24.080647 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.080476    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:24.080647 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.080489    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75ssn\" (UniqueName: \"kubernetes.io/projected/b7f0022f-3482-435d-824d-cedb3f6495cb-kube-api-access-75ssn\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:24.080647 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.080502    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:24.080647 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.080514    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f0022f-3482-435d-824d-cedb3f6495cb-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:23:24.443100 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.443064    2578 generic.go:356] "Generic (PLEG): container finished" podID="b7f0022f-3482-435d-824d-cedb3f6495cb" containerID="6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85" exitCode=0
Sep 08 15:23:24.443272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.443109    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"
Sep 08 15:23:24.443272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.443114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w" event={"ID":"b7f0022f-3482-435d-824d-cedb3f6495cb","Type":"ContainerDied","Data":"6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85"}
Sep 08 15:23:24.443272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.443162    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w" event={"ID":"b7f0022f-3482-435d-824d-cedb3f6495cb","Type":"ContainerDied","Data":"809f05ff184f4e64aa4d1301d1ca3d1eb59173ed47387c7371fe9d053d838f49"}
Sep 08 15:23:24.443272 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.443180    2578 scope.go:117] "RemoveContainer" containerID="6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85"
Sep 08 15:23:24.452280 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.452264    2578 scope.go:117] "RemoveContainer" containerID="6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85"
Sep 08 15:23:24.452625 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:23:24.452600    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85\": container with ID starting with 6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85 not found: ID does not exist" containerID="6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85"
Sep 08 15:23:24.452706 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.452630    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85"} err="failed to get container status \"6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85\": rpc error: code = NotFound desc = could not find container \"6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85\": container with ID starting with 6a68381e18e9474eefb2cc521bec3a2aff58e19b6a358371cc032f642f8c8a85 not found: ID does not exist"
Sep 08 15:23:24.464745 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.464721    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"]
Sep 08 15:23:24.470377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:24.470357    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bdd5c7/ipp-route-bdd5c7-kserve-854dd98855-s6v7w"]
Sep 08 15:23:25.832221 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:25.832189    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7f0022f-3482-435d-824d-cedb3f6495cb" path="/var/lib/kubelet/pods/b7f0022f-3482-435d-824d-cedb3f6495cb/volumes"
Sep 08 15:23:45.464180 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:45.464097    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:45.767517 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:45.767485    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:51.486660 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:51.486626    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:51.586449 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:51.586412    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:51.699458 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:51.699426    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:23:52.201908 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:23:52.201875    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:02.700632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:02.700602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:02.770713 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:02.770677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:02.834246 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:02.834216    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:03.383458 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:03.383427    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:04.834478 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:04.834446    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:04.935712 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:04.935677    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:06.808569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:06.808535    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:09.545263 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:09.545212    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:09.645772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:09.645745    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:09.863166 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:09.863073    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:11.536645 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:11.536602    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:20.807421 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:20.807390    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"]
Sep 08 15:24:20.810606 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:20.810583    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ffa941-29814675-nsd66"]
Sep 08 15:24:20.833772 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:20.833743    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"]
Sep 08 15:24:20.834015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:20.833989    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2" podUID="156d3494-a178-4015-a22b-1b98ca825d43" containerName="payload-processing" containerID="cri-o://2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73" gracePeriod=30
Sep 08 15:24:20.873392 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:20.873363    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:20.973698 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:20.973669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:21.049232 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.049198    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:21.081255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.081237    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:24:21.135269 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.135240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tqp7k\" (UniqueName: \"kubernetes.io/projected/156d3494-a178-4015-a22b-1b98ca825d43-kube-api-access-tqp7k\") pod \"156d3494-a178-4015-a22b-1b98ca825d43\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") "
Sep 08 15:24:21.135400 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.135341    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3494-a178-4015-a22b-1b98ca825d43-plugins-config-volume\") pod \"156d3494-a178-4015-a22b-1b98ca825d43\" (UID: \"156d3494-a178-4015-a22b-1b98ca825d43\") "
Sep 08 15:24:21.135695 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.135672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156d3494-a178-4015-a22b-1b98ca825d43-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "156d3494-a178-4015-a22b-1b98ca825d43" (UID: "156d3494-a178-4015-a22b-1b98ca825d43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:24:21.137435 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.137415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156d3494-a178-4015-a22b-1b98ca825d43-kube-api-access-tqp7k" (OuterVolumeSpecName: "kube-api-access-tqp7k") pod "156d3494-a178-4015-a22b-1b98ca825d43" (UID: "156d3494-a178-4015-a22b-1b98ca825d43"). InnerVolumeSpecName "kube-api-access-tqp7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:24:21.236265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.236232    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/156d3494-a178-4015-a22b-1b98ca825d43-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:21.236265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.236266    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqp7k\" (UniqueName: \"kubernetes.io/projected/156d3494-a178-4015-a22b-1b98ca825d43-kube-api-access-tqp7k\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:21.409252 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.409196    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:21.714062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.713998    2578 generic.go:356] "Generic (PLEG): container finished" podID="156d3494-a178-4015-a22b-1b98ca825d43" containerID="2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73" exitCode=0
Sep 08 15:24:21.714062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.714053    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"
Sep 08 15:24:21.714205 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.714065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2" event={"ID":"156d3494-a178-4015-a22b-1b98ca825d43","Type":"ContainerDied","Data":"2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73"}
Sep 08 15:24:21.714205 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.714109    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2" event={"ID":"156d3494-a178-4015-a22b-1b98ca825d43","Type":"ContainerDied","Data":"1edbf91dcadc8fe585e6498e2aff276f260a6f09dd86231ff82f38370d653445"}
Sep 08 15:24:21.714205 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.714133    2578 scope.go:117] "RemoveContainer" containerID="2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73"
Sep 08 15:24:21.723945 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.723927    2578 scope.go:117] "RemoveContainer" containerID="2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73"
Sep 08 15:24:21.724259 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:24:21.724236    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73\": container with ID starting with 2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73 not found: ID does not exist" containerID="2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73"
Sep 08 15:24:21.724307 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.724273    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73"} err="failed to get container status \"2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73\": rpc error: code = NotFound desc = could not find container \"2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73\": container with ID starting with 2a00d6371bd311f516d3c6089ed1dd41400bc83e8e2a68d5973c95776d7ebf73 not found: ID does not exist"
Sep 08 15:24:21.735068 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.735044    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"]
Sep 08 15:24:21.738732 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.738709    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-ffa941-78665b8977-cqhp2"]
Sep 08 15:24:21.832295 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.832273    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156d3494-a178-4015-a22b-1b98ca825d43" path="/var/lib/kubelet/pods/156d3494-a178-4015-a22b-1b98ca825d43/volumes"
Sep 08 15:24:21.832593 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:21.832548    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e61371-6a4b-48ca-96bd-95a5aabe7886" path="/var/lib/kubelet/pods/b9e61371-6a4b-48ca-96bd-95a5aabe7886/volumes"
Sep 08 15:24:22.096131 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:22.096100    2578 scope.go:117] "RemoveContainer" containerID="6c662d487d6a3feb4b33207532d289acd8b2d2b962fde01aaf20e79707d49946"
Sep 08 15:24:22.369941 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:22.369866    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:22.489391 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:22.489369    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:25.894175 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:25.894140    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:26.741282 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:26.741246    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:27.863102 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:27.863069    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:27.957446 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:27.957414    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:28.197972 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:28.197903    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:28.715299 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:28.715250    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.152718 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.152685    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.180270 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.180241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"]
Sep 08 15:24:39.180578 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.180491    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96" podUID="cca80a1b-adb6-4d4a-a968-9b597242d223" containerName="payload-processing" containerID="cri-o://2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c" gracePeriod=30
Sep 08 15:24:39.188839 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.188808    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"]
Sep 08 15:24:39.189042 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.189003    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8" podUID="2c86cb89-5596-4106-98cb-f237803a849b" containerName="payload-pre-processing" containerID="cri-o://a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff" gracePeriod=30
Sep 08 15:24:39.228833 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.228811    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.303952 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.303929    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.380073 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.379755    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.429168 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.429145    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:24:39.454794 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.454763    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.460699 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.460679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:24:39.478669 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.478482    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btkl8\" (UniqueName: \"kubernetes.io/projected/cca80a1b-adb6-4d4a-a968-9b597242d223-kube-api-access-btkl8\") pod \"cca80a1b-adb6-4d4a-a968-9b597242d223\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") "
Sep 08 15:24:39.478792 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.478696    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca80a1b-adb6-4d4a-a968-9b597242d223-plugins-config-volume\") pod \"cca80a1b-adb6-4d4a-a968-9b597242d223\" (UID: \"cca80a1b-adb6-4d4a-a968-9b597242d223\") "
Sep 08 15:24:39.479299 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.479270    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca80a1b-adb6-4d4a-a968-9b597242d223-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cca80a1b-adb6-4d4a-a968-9b597242d223" (UID: "cca80a1b-adb6-4d4a-a968-9b597242d223"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:24:39.481330 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.481300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca80a1b-adb6-4d4a-a968-9b597242d223-kube-api-access-btkl8" (OuterVolumeSpecName: "kube-api-access-btkl8") pod "cca80a1b-adb6-4d4a-a968-9b597242d223" (UID: "cca80a1b-adb6-4d4a-a968-9b597242d223"). InnerVolumeSpecName "kube-api-access-btkl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:24:39.579340 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.579318    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvfxt\" (UniqueName: \"kubernetes.io/projected/2c86cb89-5596-4106-98cb-f237803a849b-kube-api-access-pvfxt\") pod \"2c86cb89-5596-4106-98cb-f237803a849b\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") "
Sep 08 15:24:39.579439 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.579399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c86cb89-5596-4106-98cb-f237803a849b-plugins-config-volume\") pod \"2c86cb89-5596-4106-98cb-f237803a849b\" (UID: \"2c86cb89-5596-4106-98cb-f237803a849b\") "
Sep 08 15:24:39.579562 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.579548    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cca80a1b-adb6-4d4a-a968-9b597242d223-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:39.579611 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.579562    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkl8\" (UniqueName: \"kubernetes.io/projected/cca80a1b-adb6-4d4a-a968-9b597242d223-kube-api-access-btkl8\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:39.579785 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.579764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c86cb89-5596-4106-98cb-f237803a849b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c86cb89-5596-4106-98cb-f237803a849b" (UID: "2c86cb89-5596-4106-98cb-f237803a849b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:24:39.581330 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.581313    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c86cb89-5596-4106-98cb-f237803a849b-kube-api-access-pvfxt" (OuterVolumeSpecName: "kube-api-access-pvfxt") pod "2c86cb89-5596-4106-98cb-f237803a849b" (UID: "2c86cb89-5596-4106-98cb-f237803a849b"). InnerVolumeSpecName "kube-api-access-pvfxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:24:39.680899 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.680838    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfxt\" (UniqueName: \"kubernetes.io/projected/2c86cb89-5596-4106-98cb-f237803a849b-kube-api-access-pvfxt\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:39.680899 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.680857    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c86cb89-5596-4106-98cb-f237803a849b-plugins-config-volume\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:39.799468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.799437    2578 generic.go:356] "Generic (PLEG): container finished" podID="2c86cb89-5596-4106-98cb-f237803a849b" containerID="a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff" exitCode=0
Sep 08 15:24:39.799632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.799488    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"
Sep 08 15:24:39.799632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.799546    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8" event={"ID":"2c86cb89-5596-4106-98cb-f237803a849b","Type":"ContainerDied","Data":"a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff"}
Sep 08 15:24:39.799632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.799598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8" event={"ID":"2c86cb89-5596-4106-98cb-f237803a849b","Type":"ContainerDied","Data":"e56f6f117a1960ac8a6565c4fce9172b320f96811b72d4b9e822ea62f54c8b23"}
Sep 08 15:24:39.799632 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.799619    2578 scope.go:117] "RemoveContainer" containerID="a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff"
Sep 08 15:24:39.800898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.800878    2578 generic.go:356] "Generic (PLEG): container finished" podID="cca80a1b-adb6-4d4a-a968-9b597242d223" containerID="2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c" exitCode=0
Sep 08 15:24:39.800965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.800919    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"
Sep 08 15:24:39.800965 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.800950    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96" event={"ID":"cca80a1b-adb6-4d4a-a968-9b597242d223","Type":"ContainerDied","Data":"2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c"}
Sep 08 15:24:39.801047 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.800985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96" event={"ID":"cca80a1b-adb6-4d4a-a968-9b597242d223","Type":"ContainerDied","Data":"938e9f86b71a5759bf0e8f7ff28e92c2559a39ba8bc75a5386ac53bd12103e0a"}
Sep 08 15:24:39.808338 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.808319    2578 scope.go:117] "RemoveContainer" containerID="a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff"
Sep 08 15:24:39.808720 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:24:39.808614    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff\": container with ID starting with a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff not found: ID does not exist" containerID="a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff"
Sep 08 15:24:39.808720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.808640    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff"} err="failed to get container status \"a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff\": rpc error: code = NotFound desc = could not find container \"a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff\": container with ID starting with a25d0c673805b59145ebd3a7b42be7c53effec4f15d931520981f892f3e647ff not found: ID does not exist"
Sep 08 15:24:39.808720 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.808658    2578 scope.go:117] "RemoveContainer" containerID="2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c"
Sep 08 15:24:39.816296 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.816281    2578 scope.go:117] "RemoveContainer" containerID="2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c"
Sep 08 15:24:39.816564 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:24:39.816547    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c\": container with ID starting with 2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c not found: ID does not exist" containerID="2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c"
Sep 08 15:24:39.816629 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.816574    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c"} err="failed to get container status \"2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c\": rpc error: code = NotFound desc = could not find container \"2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c\": container with ID starting with 2d601057aeda803db49b4eb00e5a8fea1c363d63c6ea8c5c788a5a499ff0190c not found: ID does not exist"
Sep 08 15:24:39.824839 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.824817    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"]
Sep 08 15:24:39.831631 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.831611    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-7be222-86965ccc65-ksp96"]
Sep 08 15:24:39.832176 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.832154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:39.838895 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.838876    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"]
Sep 08 15:24:39.847545 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:39.847507    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7be222-666d8c74cd-k5lx8"]
Sep 08 15:24:41.831015 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:41.830987    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c86cb89-5596-4106-98cb-f237803a849b" path="/var/lib/kubelet/pods/2c86cb89-5596-4106-98cb-f237803a849b/volumes"
Sep 08 15:24:41.831369 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:41.831307    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cca80a1b-adb6-4d4a-a968-9b597242d223" path="/var/lib/kubelet/pods/cca80a1b-adb6-4d4a-a968-9b597242d223/volumes"
Sep 08 15:24:45.603665 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:45.603627    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:45.701143 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:45.701111    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:45.850416 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:45.850374    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:24:49.203103 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.203067    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-6lcrg"]
Sep 08 15:24:49.203495 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.203307    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg" podUID="1bd71a30-d5bb-4c2a-bb52-a54161d6494f" containerName="manager" containerID="cri-o://c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926" gracePeriod=30
Sep 08 15:24:49.450095 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.450073    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:24:49.558698 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.558660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-metrics-certs\") pod \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") "
Sep 08 15:24:49.558698 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.558707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wws7x\" (UniqueName: \"kubernetes.io/projected/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-kube-api-access-wws7x\") pod \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") "
Sep 08 15:24:49.558936 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.558907    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-webhook-cert\") pod \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\" (UID: \"1bd71a30-d5bb-4c2a-bb52-a54161d6494f\") "
Sep 08 15:24:49.561079 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.561038    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1bd71a30-d5bb-4c2a-bb52-a54161d6494f" (UID: "1bd71a30-d5bb-4c2a-bb52-a54161d6494f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:24:49.561079 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.561054    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1bd71a30-d5bb-4c2a-bb52-a54161d6494f" (UID: "1bd71a30-d5bb-4c2a-bb52-a54161d6494f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:24:49.561262 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.561147    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-kube-api-access-wws7x" (OuterVolumeSpecName: "kube-api-access-wws7x") pod "1bd71a30-d5bb-4c2a-bb52-a54161d6494f" (UID: "1bd71a30-d5bb-4c2a-bb52-a54161d6494f"). InnerVolumeSpecName "kube-api-access-wws7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:24:49.660481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.660441    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-webhook-cert\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:49.660481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.660473    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:49.660481 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.660482    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wws7x\" (UniqueName: \"kubernetes.io/projected/1bd71a30-d5bb-4c2a-bb52-a54161d6494f-kube-api-access-wws7x\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:24:49.850233 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.850203    2578 generic.go:356] "Generic (PLEG): container finished" podID="1bd71a30-d5bb-4c2a-bb52-a54161d6494f" containerID="c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926" exitCode=0
Sep 08 15:24:49.850396 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.850251    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg"
Sep 08 15:24:49.850396 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.850294    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg" event={"ID":"1bd71a30-d5bb-4c2a-bb52-a54161d6494f","Type":"ContainerDied","Data":"c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926"}
Sep 08 15:24:49.850396 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.850342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-6lcrg" event={"ID":"1bd71a30-d5bb-4c2a-bb52-a54161d6494f","Type":"ContainerDied","Data":"833b85ee87384edebec4936879eac3c24347a5e411777763268c43ee25d07f6b"}
Sep 08 15:24:49.850396 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.850365    2578 scope.go:117] "RemoveContainer" containerID="c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926"
Sep 08 15:24:49.859152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.859124    2578 scope.go:117] "RemoveContainer" containerID="c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926"
Sep 08 15:24:49.859455 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:24:49.859426    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926\": container with ID starting with c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926 not found: ID does not exist" containerID="c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926"
Sep 08 15:24:49.859607 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.859465    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926"} err="failed to get container status \"c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926\": rpc error: code = NotFound desc = could not find container \"c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926\": container with ID starting with c17d3e4343337e0b8f7f83ae6c57c1eb61895676306045d28e4679f302b21926 not found: ID does not exist"
Sep 08 15:24:49.870425 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.870396    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-6lcrg"]
Sep 08 15:24:49.874969 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:49.874944    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-6lcrg"]
Sep 08 15:24:51.832641 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:24:51.832602    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bd71a30-d5bb-4c2a-bb52-a54161d6494f" path="/var/lib/kubelet/pods/1bd71a30-d5bb-4c2a-bb52-a54161d6494f/volumes"
Sep 08 15:25:27.468621 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:27.468586    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:25:50.733334 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733303    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-7nh5d"]
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733609    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cca80a1b-adb6-4d4a-a968-9b597242d223" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cca80a1b-adb6-4d4a-a968-9b597242d223" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733631    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156d3494-a178-4015-a22b-1b98ca825d43" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733636    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="156d3494-a178-4015-a22b-1b98ca825d43" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733644    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733649    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733655    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5614deb9-a1a9-4059-8825-81c69e3de0db" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733660    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5614deb9-a1a9-4059-8825-81c69e3de0db" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733667    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bd71a30-d5bb-4c2a-bb52-a54161d6494f" containerName="manager"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733672    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bd71a30-d5bb-4c2a-bb52-a54161d6494f" containerName="manager"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733677    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c86cb89-5596-4106-98cb-f237803a849b" containerName="payload-pre-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733681    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c86cb89-5596-4106-98cb-f237803a849b" containerName="payload-pre-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733696    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7f0022f-3482-435d-824d-cedb3f6495cb" containerName="main"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733700    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7f0022f-3482-435d-824d-cedb3f6495cb" containerName="main"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733706    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a172f667-c655-4a50-95ea-8cbc080e0048" containerName="payload-pre-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733711    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a172f667-c655-4a50-95ea-8cbc080e0048" containerName="payload-pre-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cca80a1b-adb6-4d4a-a968-9b597242d223" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733778    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7f0022f-3482-435d-824d-cedb3f6495cb" containerName="main"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733784    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a172f667-c655-4a50-95ea-8cbc080e0048" containerName="payload-pre-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733789    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c86cb89-5596-4106-98cb-f237803a849b" containerName="payload-pre-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733794    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="156d3494-a178-4015-a22b-1b98ca825d43" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733801    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5614deb9-a1a9-4059-8825-81c69e3de0db" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733808    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5d5b17f-dbae-4927-9c7e-443fa6f70cf5" containerName="payload-processing"
Sep 08 15:25:50.735764 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.733812    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bd71a30-d5bb-4c2a-bb52-a54161d6494f" containerName="manager"
Sep 08 15:25:50.736636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.736620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.740562 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.740545    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 15:25:50.741498 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.741482    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 15:25:50.741652 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.741634    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dv4gq\""
Sep 08 15:25:50.741709 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.741695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 15:25:50.748255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.748234    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-7nh5d"]
Sep 08 15:25:50.874745 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.874713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-webhook-cert\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.874895 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.874776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-metrics-certs\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.874895 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.874840    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrl7\" (UniqueName: \"kubernetes.io/projected/e70887d5-bb16-4ff5-8b84-e12f3577b704-kube-api-access-lqrl7\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.975226 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.975197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrl7\" (UniqueName: \"kubernetes.io/projected/e70887d5-bb16-4ff5-8b84-e12f3577b704-kube-api-access-lqrl7\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.975386 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.975244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-webhook-cert\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.975386 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.975301    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-metrics-certs\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.977939 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.977914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-metrics-certs\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.978060 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.977974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-webhook-cert\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:50.983830 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:50.983775    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrl7\" (UniqueName: \"kubernetes.io/projected/e70887d5-bb16-4ff5-8b84-e12f3577b704-kube-api-access-lqrl7\") pod \"maas-controller-5fb695bbcb-7nh5d\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") " pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:51.047473 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:51.047439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:51.320221 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:51.320194    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-7nh5d"]
Sep 08 15:25:51.322177 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:25:51.322151    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70887d5_bb16_4ff5_8b84_e12f3577b704.slice/crio-bf2dd2e5598647ed466ba1f8a384d511bfdad54df87a7f0adf87199b42f3269f WatchSource:0}: Error finding container bf2dd2e5598647ed466ba1f8a384d511bfdad54df87a7f0adf87199b42f3269f: Status 404 returned error can't find the container with id bf2dd2e5598647ed466ba1f8a384d511bfdad54df87a7f0adf87199b42f3269f
Sep 08 15:25:51.323926 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:51.323906    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 15:25:52.123071 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:52.123037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d" event={"ID":"e70887d5-bb16-4ff5-8b84-e12f3577b704","Type":"ContainerStarted","Data":"de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308"}
Sep 08 15:25:52.123366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:52.123136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d" event={"ID":"e70887d5-bb16-4ff5-8b84-e12f3577b704","Type":"ContainerStarted","Data":"bf2dd2e5598647ed466ba1f8a384d511bfdad54df87a7f0adf87199b42f3269f"}
Sep 08 15:25:52.123366 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:52.123168    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:25:52.140753 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:25:52.140714    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d" podStartSLOduration=1.431420873 podStartE2EDuration="2.140699795s" totalImagesPullingTime="709.278922ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:25:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:25:52.13976695 +0000 UTC m=+1830.791191261" watchObservedRunningTime="2026-09-08 15:25:52.140699795 +0000 UTC m=+1830.792124108"
Sep 08 15:26:03.130200 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:03.130170    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:26:12.442798 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:12.442760    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:26:14.590983 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.590954    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-7nh5d"]
Sep 08 15:26:14.591411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.591159    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d" podUID="e70887d5-bb16-4ff5-8b84-e12f3577b704" containerName="manager" containerID="cri-o://de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308" gracePeriod=30
Sep 08 15:26:14.838152 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.838131    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:26:14.862766 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.862695    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-webhook-cert\") pod \"e70887d5-bb16-4ff5-8b84-e12f3577b704\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") "
Sep 08 15:26:14.862892 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.862799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-metrics-certs\") pod \"e70887d5-bb16-4ff5-8b84-e12f3577b704\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") "
Sep 08 15:26:14.862892 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.862831    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqrl7\" (UniqueName: \"kubernetes.io/projected/e70887d5-bb16-4ff5-8b84-e12f3577b704-kube-api-access-lqrl7\") pod \"e70887d5-bb16-4ff5-8b84-e12f3577b704\" (UID: \"e70887d5-bb16-4ff5-8b84-e12f3577b704\") "
Sep 08 15:26:14.865449 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.865421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e70887d5-bb16-4ff5-8b84-e12f3577b704" (UID: "e70887d5-bb16-4ff5-8b84-e12f3577b704"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:14.865726 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.865694    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70887d5-bb16-4ff5-8b84-e12f3577b704-kube-api-access-lqrl7" (OuterVolumeSpecName: "kube-api-access-lqrl7") pod "e70887d5-bb16-4ff5-8b84-e12f3577b704" (UID: "e70887d5-bb16-4ff5-8b84-e12f3577b704"). InnerVolumeSpecName "kube-api-access-lqrl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:26:14.865726 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.865711    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e70887d5-bb16-4ff5-8b84-e12f3577b704" (UID: "e70887d5-bb16-4ff5-8b84-e12f3577b704"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:26:14.964116 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.964090    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:26:14.964116 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.964115    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrl7\" (UniqueName: \"kubernetes.io/projected/e70887d5-bb16-4ff5-8b84-e12f3577b704-kube-api-access-lqrl7\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:26:14.964265 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:14.964125    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e70887d5-bb16-4ff5-8b84-e12f3577b704-webhook-cert\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:26:15.232178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.232115    2578 generic.go:356] "Generic (PLEG): container finished" podID="e70887d5-bb16-4ff5-8b84-e12f3577b704" containerID="de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308" exitCode=0
Sep 08 15:26:15.232178 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.232163    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d"
Sep 08 15:26:15.232306 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.232190    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d" event={"ID":"e70887d5-bb16-4ff5-8b84-e12f3577b704","Type":"ContainerDied","Data":"de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308"}
Sep 08 15:26:15.232306 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.232231    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-7nh5d" event={"ID":"e70887d5-bb16-4ff5-8b84-e12f3577b704","Type":"ContainerDied","Data":"bf2dd2e5598647ed466ba1f8a384d511bfdad54df87a7f0adf87199b42f3269f"}
Sep 08 15:26:15.232306 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.232253    2578 scope.go:117] "RemoveContainer" containerID="de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308"
Sep 08 15:26:15.247196 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.247171    2578 scope.go:117] "RemoveContainer" containerID="de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308"
Sep 08 15:26:15.247587 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:26:15.247549    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308\": container with ID starting with de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308 not found: ID does not exist" containerID="de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308"
Sep 08 15:26:15.247676 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.247603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308"} err="failed to get container status \"de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308\": rpc error: code = NotFound desc = could not find container \"de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308\": container with ID starting with de558821804a9801f729c8e4a09835dd0e24bee671c225ff9549c36b4cdcd308 not found: ID does not exist"
Sep 08 15:26:15.256274 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.256254    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-7nh5d"]
Sep 08 15:26:15.260037 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.260016    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-7nh5d"]
Sep 08 15:26:15.831766 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:15.831739    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70887d5-bb16-4ff5-8b84-e12f3577b704" path="/var/lib/kubelet/pods/e70887d5-bb16-4ff5-8b84-e12f3577b704/volumes"
Sep 08 15:26:16.043282 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.043258    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-44sps"]
Sep 08 15:26:16.043579 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.043566    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70887d5-bb16-4ff5-8b84-e12f3577b704" containerName="manager"
Sep 08 15:26:16.043637 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.043581    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70887d5-bb16-4ff5-8b84-e12f3577b704" containerName="manager"
Sep 08 15:26:16.043676 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.043648    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70887d5-bb16-4ff5-8b84-e12f3577b704" containerName="manager"
Sep 08 15:26:16.047840 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.047824    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.051435 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.051417    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 15:26:16.051573 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.051422    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 15:26:16.051573 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.051465    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 15:26:16.051573 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.051490    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dv4gq\""
Sep 08 15:26:16.056023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.056001    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-44sps"]
Sep 08 15:26:16.072377 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.072358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f446l\" (UniqueName: \"kubernetes.io/projected/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-kube-api-access-f446l\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.072472 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.072403    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-webhook-cert\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.072472 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.072427    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-metrics-certs\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.173437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.173386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f446l\" (UniqueName: \"kubernetes.io/projected/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-kube-api-access-f446l\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.173437 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.173430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-webhook-cert\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.173591 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.173445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-metrics-certs\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.175998 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.175974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-metrics-certs\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.176076 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.176031    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-webhook-cert\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.181155 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.181133    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f446l\" (UniqueName: \"kubernetes.io/projected/d21771ad-41c7-4f89-8c20-4a7c7128d0e5-kube-api-access-f446l\") pod \"maas-controller-5fb695bbcb-44sps\" (UID: \"d21771ad-41c7-4f89-8c20-4a7c7128d0e5\") " pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.359645 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.359622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:16.633717 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.633687    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5fb695bbcb-44sps"]
Sep 08 15:26:16.636134 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:26:16.636106    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21771ad_41c7_4f89_8c20_4a7c7128d0e5.slice/crio-e5104a2df7f439849c946da05a4cc7be207c35133064d68d0e5abc4a287acbfa WatchSource:0}: Error finding container e5104a2df7f439849c946da05a4cc7be207c35133064d68d0e5abc4a287acbfa: Status 404 returned error can't find the container with id e5104a2df7f439849c946da05a4cc7be207c35133064d68d0e5abc4a287acbfa
Sep 08 15:26:16.707901 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:16.707873    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:26:17.243057 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:17.243033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-44sps" event={"ID":"d21771ad-41c7-4f89-8c20-4a7c7128d0e5","Type":"ContainerStarted","Data":"e5104a2df7f439849c946da05a4cc7be207c35133064d68d0e5abc4a287acbfa"}
Sep 08 15:26:18.248461 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:18.248427    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5fb695bbcb-44sps" event={"ID":"d21771ad-41c7-4f89-8c20-4a7c7128d0e5","Type":"ContainerStarted","Data":"d788cf1c572b56a3fab5b4453a169ef195906ad5dd8b407fb0dcf4f06050ef7c"}
Sep 08 15:26:18.248911 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:18.248558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:18.266654 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:18.266614    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5fb695bbcb-44sps" podStartSLOduration=1.6812190070000002 podStartE2EDuration="2.266600099s" totalImagesPullingTime="585.381092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:26:18.265241016 +0000 UTC m=+1856.916665326" watchObservedRunningTime="2026-09-08 15:26:18.266600099 +0000 UTC m=+1856.918024407"
Sep 08 15:26:29.253825 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:29.253789    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5fb695bbcb-44sps"
Sep 08 15:26:42.463077 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:26:42.463035    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:29:17.213292 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.213262    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"]
Sep 08 15:29:17.216301 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.216285    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.220194 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.220174    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-0572b5a6-kserve-self-signed-certs\""
Sep 08 15:29:17.229411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.229391    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"]
Sep 08 15:29:17.274023 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.273999    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-home\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.274134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.274065    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-tmp-dir\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.274134 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.274114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzvgn\" (UniqueName: \"kubernetes.io/projected/7b478588-7670-4d64-9bd0-4bed6bec1245-kube-api-access-vzvgn\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.274279 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.274210    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-model-cache\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.274279 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.274236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-dshm\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.274394 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.274331    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b478588-7670-4d64-9bd0-4bed6bec1245-tls-certs\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375151 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-model-cache\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-dshm\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375180    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b478588-7670-4d64-9bd0-4bed6bec1245-tls-certs\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375222    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-home\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-tmp-dir\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375264 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375252    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzvgn\" (UniqueName: \"kubernetes.io/projected/7b478588-7670-4d64-9bd0-4bed6bec1245-kube-api-access-vzvgn\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375561 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375514    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-model-cache\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375640 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375620    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-tmp-dir\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.375724 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.375707    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-home\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.377557 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.377517    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-dshm\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.378088 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.378065    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b478588-7670-4d64-9bd0-4bed6bec1245-tls-certs\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.387837 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.387811    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzvgn\" (UniqueName: \"kubernetes.io/projected/7b478588-7670-4d64-9bd0-4bed6bec1245-kube-api-access-vzvgn\") pod \"e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") " pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.525363 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.525332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:17.566821 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.566781    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:29:17.798972 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:17.798947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"]
Sep 08 15:29:17.801323 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:29:17.801297    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b478588_7670_4d64_9bd0_4bed6bec1245.slice/crio-a122137a48476d9c2db59e30b926fe033ec02ef9359d9ab78c36e9698a2b64fd WatchSource:0}: Error finding container a122137a48476d9c2db59e30b926fe033ec02ef9359d9ab78c36e9698a2b64fd: Status 404 returned error can't find the container with id a122137a48476d9c2db59e30b926fe033ec02ef9359d9ab78c36e9698a2b64fd
Sep 08 15:29:18.011978 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:18.011945    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g" event={"ID":"7b478588-7670-4d64-9bd0-4bed6bec1245","Type":"ContainerStarted","Data":"7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80"}
Sep 08 15:29:18.012133 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:18.011991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g" event={"ID":"7b478588-7670-4d64-9bd0-4bed6bec1245","Type":"ContainerStarted","Data":"a122137a48476d9c2db59e30b926fe033ec02ef9359d9ab78c36e9698a2b64fd"}
Sep 08 15:29:18.038018 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:18.037973    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g" podStartSLOduration=1.037959646 podStartE2EDuration="1.037959646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:29:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:29:18.036321524 +0000 UTC m=+2036.687745823" watchObservedRunningTime="2026-09-08 15:29:18.037959646 +0000 UTC m=+2036.689383955"
Sep 08 15:29:27.526232 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:27.526195    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:27.526232 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:27.526241    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:27.538496 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:27.538477    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:28.064198 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:28.064166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:29:31.419559 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:31.419504    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:29:48.438184 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:48.438097    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"]
Sep 08 15:29:48.459552 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:48.459502    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:29:49.201221 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:29:49.201192    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:30:03.458306 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.458252    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g" podUID="7b478588-7670-4d64-9bd0-4bed6bec1245" containerName="main" containerID="cri-o://7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80" gracePeriod=45
Sep 08 15:30:03.698628 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.698607    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:30:03.829243 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829215    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-home\") pod \"7b478588-7670-4d64-9bd0-4bed6bec1245\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") "
Sep 08 15:30:03.829407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829267    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-tmp-dir\") pod \"7b478588-7670-4d64-9bd0-4bed6bec1245\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") "
Sep 08 15:30:03.829407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829292    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-model-cache\") pod \"7b478588-7670-4d64-9bd0-4bed6bec1245\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") "
Sep 08 15:30:03.829407 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-dshm\") pod \"7b478588-7670-4d64-9bd0-4bed6bec1245\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") "
Sep 08 15:30:03.829602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829420    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b478588-7670-4d64-9bd0-4bed6bec1245-tls-certs\") pod \"7b478588-7670-4d64-9bd0-4bed6bec1245\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") "
Sep 08 15:30:03.829602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829445    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-home" (OuterVolumeSpecName: "home") pod "7b478588-7670-4d64-9bd0-4bed6bec1245" (UID: "7b478588-7670-4d64-9bd0-4bed6bec1245"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:30:03.829602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzvgn\" (UniqueName: \"kubernetes.io/projected/7b478588-7670-4d64-9bd0-4bed6bec1245-kube-api-access-vzvgn\") pod \"7b478588-7670-4d64-9bd0-4bed6bec1245\" (UID: \"7b478588-7670-4d64-9bd0-4bed6bec1245\") "
Sep 08 15:30:03.829602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829556    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b478588-7670-4d64-9bd0-4bed6bec1245" (UID: "7b478588-7670-4d64-9bd0-4bed6bec1245"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:30:03.829809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829669    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b478588-7670-4d64-9bd0-4bed6bec1245" (UID: "7b478588-7670-4d64-9bd0-4bed6bec1245"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:30:03.829809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829788    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-home\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:03.829809 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829805    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-tmp-dir\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:03.829936 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.829820    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-model-cache\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:03.831946 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.831915    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b478588-7670-4d64-9bd0-4bed6bec1245-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b478588-7670-4d64-9bd0-4bed6bec1245" (UID: "7b478588-7670-4d64-9bd0-4bed6bec1245"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:30:03.832113 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.832091    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-dshm" (OuterVolumeSpecName: "dshm") pod "7b478588-7670-4d64-9bd0-4bed6bec1245" (UID: "7b478588-7670-4d64-9bd0-4bed6bec1245"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 15:30:03.832255 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.832236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b478588-7670-4d64-9bd0-4bed6bec1245-kube-api-access-vzvgn" (OuterVolumeSpecName: "kube-api-access-vzvgn") pod "7b478588-7670-4d64-9bd0-4bed6bec1245" (UID: "7b478588-7670-4d64-9bd0-4bed6bec1245"). InnerVolumeSpecName "kube-api-access-vzvgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:30:03.930512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.930486    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b478588-7670-4d64-9bd0-4bed6bec1245-dshm\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:03.930512 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.930509    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b478588-7670-4d64-9bd0-4bed6bec1245-tls-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:03.930675 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:03.930549    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzvgn\" (UniqueName: \"kubernetes.io/projected/7b478588-7670-4d64-9bd0-4bed6bec1245-kube-api-access-vzvgn\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:04.212343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.212280    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b478588-7670-4d64-9bd0-4bed6bec1245" containerID="7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80" exitCode=0
Sep 08 15:30:04.212343 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.212330    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"
Sep 08 15:30:04.212494 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.212363    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g" event={"ID":"7b478588-7670-4d64-9bd0-4bed6bec1245","Type":"ContainerDied","Data":"7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80"}
Sep 08 15:30:04.212494 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.212402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g" event={"ID":"7b478588-7670-4d64-9bd0-4bed6bec1245","Type":"ContainerDied","Data":"a122137a48476d9c2db59e30b926fe033ec02ef9359d9ab78c36e9698a2b64fd"}
Sep 08 15:30:04.212494 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.212418    2578 scope.go:117] "RemoveContainer" containerID="7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80"
Sep 08 15:30:04.223177 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.223161    2578 scope.go:117] "RemoveContainer" containerID="7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80"
Sep 08 15:30:04.223498 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:30:04.223480    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80\": container with ID starting with 7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80 not found: ID does not exist" containerID="7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80"
Sep 08 15:30:04.223595 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.223508    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80"} err="failed to get container status \"7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80\": rpc error: code = NotFound desc = could not find container \"7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80\": container with ID starting with 7886457a61dfbc67b6972b1e4a574f903823443d99d51ff0bba95edd3eecbd80 not found: ID does not exist"
Sep 08 15:30:04.228315 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.228290    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"]
Sep 08 15:30:04.232860 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:04.232837    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-0572b5a6-kserve-68788d646c-gv84g"]
Sep 08 15:30:05.830891 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:05.830858    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b478588-7670-4d64-9bd0-4bed6bec1245" path="/var/lib/kubelet/pods/7b478588-7670-4d64-9bd0-4bed6bec1245/volumes"
Sep 08 15:30:14.812233 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.812196    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"]
Sep 08 15:30:14.812869 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.812650    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b478588-7670-4d64-9bd0-4bed6bec1245" containerName="main"
Sep 08 15:30:14.812869 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.812664    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b478588-7670-4d64-9bd0-4bed6bec1245" containerName="main"
Sep 08 15:30:14.812869 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.812726    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b478588-7670-4d64-9bd0-4bed6bec1245" containerName="main"
Sep 08 15:30:14.817319 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.817294    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:14.826423 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.826399    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"]
Sep 08 15:30:14.913631 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.913600    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpfp2\" (UniqueName: \"kubernetes.io/projected/db015e1b-86cd-4dac-b88d-c93482131f8d-kube-api-access-wpfp2\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:14.913631 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.913632    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db015e1b-86cd-4dac-b88d-c93482131f8d-openshift-service-ca\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:14.913819 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.913654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db015e1b-86cd-4dac-b88d-c93482131f8d-maas-api-tls\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:14.913819 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:14.913745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db015e1b-86cd-4dac-b88d-c93482131f8d-metrics-certs\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.014549 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.014492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpfp2\" (UniqueName: \"kubernetes.io/projected/db015e1b-86cd-4dac-b88d-c93482131f8d-kube-api-access-wpfp2\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.014740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.014556    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db015e1b-86cd-4dac-b88d-c93482131f8d-openshift-service-ca\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.014740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.014587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db015e1b-86cd-4dac-b88d-c93482131f8d-maas-api-tls\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.014740 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.014639    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db015e1b-86cd-4dac-b88d-c93482131f8d-metrics-certs\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.015219 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.015198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db015e1b-86cd-4dac-b88d-c93482131f8d-openshift-service-ca\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.017177 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.017157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db015e1b-86cd-4dac-b88d-c93482131f8d-metrics-certs\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.017259 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.017230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db015e1b-86cd-4dac-b88d-c93482131f8d-maas-api-tls\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.022952 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.022932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpfp2\" (UniqueName: \"kubernetes.io/projected/db015e1b-86cd-4dac-b88d-c93482131f8d-kube-api-access-wpfp2\") pod \"maas-api-5d8898d46d-vfsgk\" (UID: \"db015e1b-86cd-4dac-b88d-c93482131f8d\") " pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.128650 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.128569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:15.400482 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:15.400458    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"]
Sep 08 15:30:15.402628 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:30:15.402597    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb015e1b_86cd_4dac_b88d_c93482131f8d.slice/crio-14c8c535d608ea2d14429e1ec134461cb53448189bbbf83294c542f4bbf97cb2 WatchSource:0}: Error finding container 14c8c535d608ea2d14429e1ec134461cb53448189bbbf83294c542f4bbf97cb2: Status 404 returned error can't find the container with id 14c8c535d608ea2d14429e1ec134461cb53448189bbbf83294c542f4bbf97cb2
Sep 08 15:30:16.275898 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:16.275859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk" event={"ID":"db015e1b-86cd-4dac-b88d-c93482131f8d","Type":"ContainerStarted","Data":"8ec16d33c171f5a7910889dcd323a2192c42fc792194b29156ceed1b742ea4ce"}
Sep 08 15:30:16.276446 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:16.275920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk" event={"ID":"db015e1b-86cd-4dac-b88d-c93482131f8d","Type":"ContainerStarted","Data":"14c8c535d608ea2d14429e1ec134461cb53448189bbbf83294c542f4bbf97cb2"}
Sep 08 15:30:16.276446 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:16.275961    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:16.296311 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:16.296264    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk" podStartSLOduration=1.8920035990000001 podStartE2EDuration="2.296248119s" totalImagesPullingTime="404.24452ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 15:30:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:30:16.295100616 +0000 UTC m=+2094.946524925" watchObservedRunningTime="2026-09-08 15:30:16.296248119 +0000 UTC m=+2094.947672429"
Sep 08 15:30:22.285869 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:22.285841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d8898d46d-vfsgk"
Sep 08 15:30:22.343636 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:22.343606    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"]
Sep 08 15:30:22.343858 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:22.343830    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr" podUID="bb5a9c75-92d5-4d5f-89b1-eb563eec232d" containerName="maas-api" containerID="cri-o://b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851" gracePeriod=30
Sep 08 15:30:23.579390 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.579367    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:30:23.685111 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.685087    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-openshift-service-ca\") pod \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") "
Sep 08 15:30:23.685249 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.685122    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-metrics-certs\") pod \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") "
Sep 08 15:30:23.685249 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.685149    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-maas-api-tls\") pod \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") "
Sep 08 15:30:23.685249 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.685240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjzzz\" (UniqueName: \"kubernetes.io/projected/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-kube-api-access-wjzzz\") pod \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\" (UID: \"bb5a9c75-92d5-4d5f-89b1-eb563eec232d\") "
Sep 08 15:30:23.685479 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.685451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bb5a9c75-92d5-4d5f-89b1-eb563eec232d" (UID: "bb5a9c75-92d5-4d5f-89b1-eb563eec232d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 15:30:23.687315 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.687293    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bb5a9c75-92d5-4d5f-89b1-eb563eec232d" (UID: "bb5a9c75-92d5-4d5f-89b1-eb563eec232d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:30:23.687426 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.687385    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bb5a9c75-92d5-4d5f-89b1-eb563eec232d" (UID: "bb5a9c75-92d5-4d5f-89b1-eb563eec232d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 15:30:23.687472 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.687421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-kube-api-access-wjzzz" (OuterVolumeSpecName: "kube-api-access-wjzzz") pod "bb5a9c75-92d5-4d5f-89b1-eb563eec232d" (UID: "bb5a9c75-92d5-4d5f-89b1-eb563eec232d"). InnerVolumeSpecName "kube-api-access-wjzzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 15:30:23.786399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.786378    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-openshift-service-ca\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:23.786399 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.786400    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-metrics-certs\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:23.786569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.786409    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-maas-api-tls\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:23.786569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:23.786418    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjzzz\" (UniqueName: \"kubernetes.io/projected/bb5a9c75-92d5-4d5f-89b1-eb563eec232d-kube-api-access-wjzzz\") on node \"ip-10-0-136-157.ec2.internal\" DevicePath \"\""
Sep 08 15:30:24.311464 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.311429    2578 generic.go:356] "Generic (PLEG): container finished" podID="bb5a9c75-92d5-4d5f-89b1-eb563eec232d" containerID="b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851" exitCode=0
Sep 08 15:30:24.311665 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.311475    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"
Sep 08 15:30:24.311665 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.311494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr" event={"ID":"bb5a9c75-92d5-4d5f-89b1-eb563eec232d","Type":"ContainerDied","Data":"b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851"}
Sep 08 15:30:24.311665 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.311560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr" event={"ID":"bb5a9c75-92d5-4d5f-89b1-eb563eec232d","Type":"ContainerDied","Data":"dab1f1327c277e7fab86300ddd0e528dfeeb84e8770dd6fe91d21de3d8b26084"}
Sep 08 15:30:24.311665 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.311581    2578 scope.go:117] "RemoveContainer" containerID="b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851"
Sep 08 15:30:24.320124 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.320106    2578 scope.go:117] "RemoveContainer" containerID="b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851"
Sep 08 15:30:24.320385 ip-10-0-136-157 kubenswrapper[2578]: E0908 15:30:24.320364    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851\": container with ID starting with b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851 not found: ID does not exist" containerID="b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851"
Sep 08 15:30:24.320439 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.320393    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851"} err="failed to get container status \"b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851\": rpc error: code = NotFound desc = could not find container \"b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851\": container with ID starting with b5f62ecc02c2aec8f94f2ac898934d607cfab972bb4a6be93cadc2cd4d367851 not found: ID does not exist"
Sep 08 15:30:24.338448 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.338424    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"]
Sep 08 15:30:24.342531 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:24.342500    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-54bb5c4896-xpxqr"]
Sep 08 15:30:25.831503 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:25.831470    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb5a9c75-92d5-4d5f-89b1-eb563eec232d" path="/var/lib/kubelet/pods/bb5a9c75-92d5-4d5f-89b1-eb563eec232d/volumes"
Sep 08 15:30:39.841897 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:39.841867    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-645df54cd8-jx685_c4e14ee6-fe20-4687-9525-e9d7e411b745/authorino/0.log"
Sep 08 15:30:46.352511 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:46.352478    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb695bbcb-44sps_d21771ad-41c7-4f89-8c20-4a7c7128d0e5/manager/0.log"
Sep 08 15:30:46.931623 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:46.931596    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b574fc88f-6wpwh_aced14fe-726f-419c-91a3-eb281d7b3cb7/copy-manifests/0.log"
Sep 08 15:30:46.939062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:46.939039    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b574fc88f-6wpwh_aced14fe-726f-419c-91a3-eb281d7b3cb7/manager/0.log"
Sep 08 15:30:47.562678 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:47.562644    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d8898d46d-vfsgk_db015e1b-86cd-4dac-b88d-c93482131f8d/maas-api/0.log"
Sep 08 15:30:47.683397 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:47.683369    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814675-xx6pl_71fbd6c9-1a51-4b11-bf42-c3f4d8e618c5/cleanup/0.log"
Sep 08 15:30:48.823906 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:48.823870    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-645df54cd8-jx685_c4e14ee6-fe20-4687-9525-e9d7e411b745/authorino/0.log"
Sep 08 15:30:49.235062 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:49.234980    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-bflpr_881b1609-d5f9-4823-a466-e76a62ffdc62/kuadrant-console-plugin/0.log"
Sep 08 15:30:50.419154 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:50.419126    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ff8c49fcd-pncgn_289d27f9-9d44-4881-ae48-bc4814ab9301/kube-auth-proxy/0.log"
Sep 08 15:30:50.950666 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:50.950631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-556b5cd57d-klghc_593351b6-a625-467e-98e2-acbb80c3096d/router/0.log"
Sep 08 15:30:51.464884 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:51.464851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-zz9j6_77e96791-e9d1-4fa6-b8e9-0d82c8a2ab20/main/0.log"
Sep 08 15:30:51.744602 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:51.744488    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qcdpt_9052627c-ec6d-4133-add5-c4d3bf33f324/main/0.log"
Sep 08 15:30:52.020782 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:52.020736    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-k4cl2_6f633d2f-f332-4f43-90bf-a51a9ddca42c/main/0.log"
Sep 08 15:30:52.156427 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:52.156400    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-k2clf_70d5e28c-117a-4aeb-a036-c552b5886f92/main/0.log"
Sep 08 15:30:58.131905 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:58.131862    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:30:58.288034 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:30:58.288003    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:31:01.457451 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:01.457413    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pdmng_68ff52bf-57a8-4714-a16b-b047539508d3/global-pull-secret-syncer/0.log"
Sep 08 15:31:01.522749 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:01.522721    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log"
Sep 08 15:31:01.641552 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:01.641507    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-157.ec2.internal_e1ff1a9c0e7f9107b441cc843400901d/haproxy/0.log"
Sep 08 15:31:06.241101 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:06.241068    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-645df54cd8-jx685_c4e14ee6-fe20-4687-9525-e9d7e411b745/authorino/0.log"
Sep 08 15:31:06.326663 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:06.326643    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-bflpr_881b1609-d5f9-4823-a466-e76a62ffdc62/kuadrant-console-plugin/0.log"
Sep 08 15:31:08.242342 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:08.242314    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gfxpg_9d357714-b7b0-4400-8a16-4957188c1380/kube-state-metrics/0.log"
Sep 08 15:31:08.261633 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:08.261601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gfxpg_9d357714-b7b0-4400-8a16-4957188c1380/kube-rbac-proxy-main/0.log"
Sep 08 15:31:08.286935 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:08.286916    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-gfxpg_9d357714-b7b0-4400-8a16-4957188c1380/kube-rbac-proxy-self/0.log"
Sep 08 15:31:08.594427 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:08.594405    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgw4l_6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f/node-exporter/0.log"
Sep 08 15:31:08.631619 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:08.631598    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgw4l_6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f/kube-rbac-proxy/0.log"
Sep 08 15:31:08.674904 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:08.674881    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgw4l_6443d6bc-b1ce-4c83-ac02-b9ec5e181c7f/init-textfile/0.log"
Sep 08 15:31:09.118551 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:09.118490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6947fb877c-m6t8w_a69aa81c-fbe3-41f7-872b-c0b48851ea5e/telemeter-client/0.log"
Sep 08 15:31:09.140974 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:09.140951    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6947fb877c-m6t8w_a69aa81c-fbe3-41f7-872b-c0b48851ea5e/reload/0.log"
Sep 08 15:31:09.163216 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:09.163191    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6947fb877c-m6t8w_a69aa81c-fbe3-41f7-872b-c0b48851ea5e/kube-rbac-proxy/0.log"
Sep 08 15:31:11.628254 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.628221    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"]
Sep 08 15:31:11.628627 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.628562    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5a9c75-92d5-4d5f-89b1-eb563eec232d" containerName="maas-api"
Sep 08 15:31:11.628627 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.628575    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5a9c75-92d5-4d5f-89b1-eb563eec232d" containerName="maas-api"
Sep 08 15:31:11.628701 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.628636    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5a9c75-92d5-4d5f-89b1-eb563eec232d" containerName="maas-api"
Sep 08 15:31:11.631415 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.631399    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.634505 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.634486    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5z7hh\"/\"kube-root-ca.crt\""
Sep 08 15:31:11.634759 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.634740    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5z7hh\"/\"openshift-service-ca.crt\""
Sep 08 15:31:11.635736 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.635718    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5z7hh\"/\"default-dockercfg-8ph62\""
Sep 08 15:31:11.644074 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.644051    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"]
Sep 08 15:31:11.759341 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.759318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-proc\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.759434 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.759351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-podres\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.759434 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.759390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-lib-modules\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.759506 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.759453    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6ssh\" (UniqueName: \"kubernetes.io/projected/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-kube-api-access-z6ssh\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.759568 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.759504    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-sys\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.860797 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860773    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-lib-modules\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.860928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6ssh\" (UniqueName: \"kubernetes.io/projected/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-kube-api-access-z6ssh\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.860928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-sys\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.860928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-proc\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.860928 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-podres\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.861164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-sys\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.861164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.860977    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-lib-modules\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.861164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.861010    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-podres\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.861164 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.861009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-proc\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.870411 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.870386    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6ssh\" (UniqueName: \"kubernetes.io/projected/8504c5e5-b2b0-46ea-8fcc-57c7cf13709e-kube-api-access-z6ssh\") pod \"perf-node-gather-daemonset-jhscb\" (UID: \"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e\") " pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:11.941509 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:11.941458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:12.215290 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:12.215265    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"]
Sep 08 15:31:12.217758 ip-10-0-136-157 kubenswrapper[2578]: W0908 15:31:12.217722    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8504c5e5_b2b0_46ea_8fcc_57c7cf13709e.slice/crio-cada5de42de3079f6c5bf4d090cb77a383717a2ed84c7fc96fba29f01fe9c9fe WatchSource:0}: Error finding container cada5de42de3079f6c5bf4d090cb77a383717a2ed84c7fc96fba29f01fe9c9fe: Status 404 returned error can't find the container with id cada5de42de3079f6c5bf4d090cb77a383717a2ed84c7fc96fba29f01fe9c9fe
Sep 08 15:31:12.512301 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:12.512269    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb" event={"ID":"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e","Type":"ContainerStarted","Data":"83de742405533025fa88e2a294ad2bd4bd6c6f721e7a63d6b636c83868ad845a"}
Sep 08 15:31:12.512468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:12.512317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb" event={"ID":"8504c5e5-b2b0-46ea-8fcc-57c7cf13709e","Type":"ContainerStarted","Data":"cada5de42de3079f6c5bf4d090cb77a383717a2ed84c7fc96fba29f01fe9c9fe"}
Sep 08 15:31:12.512468 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:12.512422    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:12.537644 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:12.532739    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb" podStartSLOduration=1.5327228179999999 podStartE2EDuration="1.532722818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 15:31:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 15:31:12.530813036 +0000 UTC m=+2151.182237345" watchObservedRunningTime="2026-09-08 15:31:12.532722818 +0000 UTC m=+2151.184147127"
Sep 08 15:31:13.100261 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:13.100211    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:31:13.170897 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:13.170868    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:31:13.991660 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:13.991622    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgn5w_aff4ee34-7afb-49df-9b72-ef2e90a39866/dns/0.log"
Sep 08 15:31:14.030831 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:14.030809    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zgn5w_aff4ee34-7afb-49df-9b72-ef2e90a39866/kube-rbac-proxy/0.log"
Sep 08 15:31:14.062251 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:14.062226    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dvvmx_a24eeded-2925-4be8-b7f6-1ffa604cce09/dns-node-resolver/0.log"
Sep 08 15:31:14.771960 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:14.771936    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z5wzq_4ce3e904-ba74-4c51-8f1f-e6d5de2129d6/node-ca/0.log"
Sep 08 15:31:16.172660 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:16.172631    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ff8c49fcd-pncgn_289d27f9-9d44-4881-ae48-bc4814ab9301/kube-auth-proxy/0.log"
Sep 08 15:31:16.363807 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:16.363783    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-556b5cd57d-klghc_593351b6-a625-467e-98e2-acbb80c3096d/router/0.log"
Sep 08 15:31:17.323227 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:17.323200    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9rlmc_bc8d39ce-77a7-421e-8789-cd4f0173f426/serve-healthcheck-canary/0.log"
Sep 08 15:31:18.228256 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:18.228222    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bq2g2_eb0f489a-6369-42fb-8518-f1ac77cfbcb5/kube-rbac-proxy/0.log"
Sep 08 15:31:18.259260 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:18.259228    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bq2g2_eb0f489a-6369-42fb-8518-f1ac77cfbcb5/exporter/0.log"
Sep 08 15:31:18.290441 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:18.290421    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bq2g2_eb0f489a-6369-42fb-8518-f1ac77cfbcb5/extractor/0.log"
Sep 08 15:31:18.528761 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:18.528739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5z7hh/perf-node-gather-daemonset-jhscb"
Sep 08 15:31:21.222877 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:21.222851    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5fb695bbcb-44sps_d21771ad-41c7-4f89-8c20-4a7c7128d0e5/manager/0.log"
Sep 08 15:31:21.875106 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:21.875074    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b574fc88f-6wpwh_aced14fe-726f-419c-91a3-eb281d7b3cb7/manager/0.log"
Sep 08 15:31:21.976693 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:21.976668    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b574fc88f-6wpwh_aced14fe-726f-419c-91a3-eb281d7b3cb7/copy-manifests/0.log"
Sep 08 15:31:22.423254 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:22.423216    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ff8c49fcd-pncgn_289d27f9-9d44-4881-ae48-bc4814ab9301/kube-auth-proxy/0.log"
Sep 08 15:31:22.724079 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:22.724021    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-556b5cd57d-klghc_593351b6-a625-467e-98e2-acbb80c3096d/router/0.log"
Sep 08 15:31:23.665055 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:23.664976    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-pll64_cb533fb6-ecee-4d0c-bb67-508a0b38e28e/konnectivity-agent/0.log" limit=1024
Sep 08 15:31:23.966375 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:23.966351    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c458b45b4-2h89t_472997ed-6272-40bf-bcce-6081a7240326/manager/0.log"
Sep 08 15:31:24.902759 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:24.902728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5ff8c49fcd-pncgn_289d27f9-9d44-4881-ae48-bc4814ab9301/kube-auth-proxy/0.log"
Sep 08 15:31:25.154910 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:25.154830    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-556b5cd57d-klghc_593351b6-a625-467e-98e2-acbb80c3096d/router/0.log"
Sep 08 15:31:30.582207 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:30.582176    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-nkfdj_db484c3f-cbab-423d-9ad4-d43d17f5ac49/migrator/0.log"
Sep 08 15:31:30.633569 ip-10-0-136-157 kubenswrapper[2578]: I0908 15:31:30.633545    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-nkfdj_db484c3f-cbab-423d-9ad4-d43d17f5ac49/graceful-termination/0.log"